Data Collection for a specific compute resource was failing on a given endpoint in vRealize Automation 7.x
The exception was as below
2020-05-14T09:31:30.772Z <<AGENTNODE>> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6016"] [sub-thread-Id="32" context="" token=""] Error retrieving properties in task 26394140: System.Web.Services.Protocols.SoapException:
Required property obj is missing from data object of type ObjectSpec
while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectSpec
at line 1, column 717Data Collection for other compute resources on the same endpoint was working as expected.
So why wouldn't it work only for one specific compute resource was the question
Went into my lab and then thought of checking the behavior to compare working and the non-working one
Usecase was to perform an inventory data collection on a selected compute resource
Here's the snippet from a working instance
2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Info" thread="6276"] [sub-thread-Id="23" context="" token=""] Starting : Processing Workitem ID [e779099f-b1df-4a31-99f9-1169feb35272] [inventory]
2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.Hostname=primary
2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.ParentIdentity=premvc.prem.com/production/host/primary
2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Name=primary
2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Identity=premvc.prem.com/production/host/primary
2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Endpoint0=10.109.10.84
2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Endpoint1=10.109.10.85
2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Endpoint2=10.109.10.86
2020-05-18T04:46:14.787Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6276"] [sub-thread-Id="23" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.Name=inventory
2020-05-18T04:46:16.412Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetHosts: 1631 milliseconds
2020-05-18T04:46:16.537Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetClusters: 123 milliseconds
2020-05-18T04:46:16.568Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetNetworks: 27 milliseconds
2020-05-18T04:46:16.599Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetResourcePools: 28 milliseconds
2020-05-18T04:46:16.631Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetStoragePools: 34 milliseconds
2020-05-18T04:46:16.631Z SEVENIAAS vcac: [component="iaas:VRMAgent.exe" priority="Trace" thread="6276"] [sub-thread-Id="23" context="" token=""] GetTemplates: 0 millisecondsAs you can see in the above snippet after it fetches VirtualMachine.Admin.Name=inventory it goes ahead and then gets the information of the inventory like GetHosts , GetClusters, GetNetworks, GetResourcePools, and GetStoragePools, etc...
If we check the same on a non-working instance
2020-05-14T09:31:30.631Z <<AGENTNODE>> vcac: [component="iaas:VRMAgent.exe" priority="Info" thread="2064"] [sub-thread-Id="18" context="" token=""] Starting : Processing Workitem ID [38e0898c-4f0a-4f66-bbf0-368fff7c82d1] [inventory]
2020-05-14T09:31:30.631Z <<AGENTNODE>> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.Hostname=<<COMPUTEREOSURCENAME>>
2020-05-14T09:31:30.631Z <<AGENTNODE>> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.ParentIdentity=<<VCENTERNAME>><<<<DATACENTERNAME>>/host/<<COMPUTEREOSURCENAME>>
2020-05-14T09:31:30.631Z <<AGENTNODE>> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Name=<<COMPUTEREOSURCENAME>>
2020-05-14T09:31:30.631Z <<AGENTNODE>> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Identity=<<VCENTERNAME>>/<<DATACENTERNAME>>/host/<<COMPUTEREOSURCENAME>>
2020-05-14T09:31:30.631Z <<AGENTNODE>> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.ManagementEndpoint.Endpoint0=<<COMPUTEREOSURCENAME>>
2020-05-14T09:31:30.631Z <<AGENTNODE>> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="2064"] [sub-thread-Id="18" context="" token=""] [[inventory]] [inventory] VirtualMachine.Admin.Name=inventory
2020-05-14T09:31:30.772Z <<AGENTNODE>> vcac: [component="iaas:VRMAgent.exe" priority="Debug" thread="6016"] [sub-thread-Id="32" context="" token=""] Error retrieving properties in task 26394140: System.Web.Services.Protocols.SoapException:
Required property obj is missing from data object of type ObjectSpec
while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectSpec
at line 1, column 717
while parsing property "objectSet" of static type ArrayOfObjectSpec
while parsing serialized DataObject of type vmodl.query.PropertyCollector.FilterSpec
at line 1, column 315
while parsing call information for method RetrievePropertiesEx
at line 1, column 218
while parsing SOAP body
at line 1, column 207
while parsing SOAP envelope
at line 1, column 38
while parsing HTTP request for method retrievePropertiesEx
on object of type vmodl.query.PropertyCollector
at line 1, column 0
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult)
at VMware.vSphere.VimService.EndRetrievePropertiesEx(IAsyncResult asyncResult)
at Vmware.VSphereInterface.RetryRequestWrapper.Execute[T](Func`1 function)If you look closely and compare it, it's clearly failing to perform retrievePropertiesEx which does your GetHosts, GetClusters, GetNetworks, GetResourcePools, and GetStoragePools
So clearly the problem is not on vRealize Automation end, it's more on vSphere.
When we went ahead and verified this particular cluster / compute resource where data collection was failing does not even exist on vCenter.
Which clearly explains the exception
Required property obj is missing from data object of type ObjectSpec
Take-Aways
The problem does not always have to be on vRealize Automation end, it can be elsewhere.
If we have integrated a vCenter as an endpoint when any changes are being made on the endpoint directly as a best practice one has to inform your vRealize Automation admins.
This would ensure before off-boarding a cluster on endpoint proper precautions are taken to offboard the same compute resource from the vRA perspective as well.
Komentarji