NO , Virtual Machines which are / were being onboarded will never be deleted automatically. Onboarding feature in vRealize Automation does not have intelligence or capability to delete the virtual machines on it's own
I was working on one such scenario where users were claiming that the machines which were being onboarded were deleted. Not one or two but hundreds.
This was a good case-study to understand what really happened in the background
Here goes he approach during my investigation and how we found out who did it
Ideal Onboarding Procedure
These are the two phases which constitute onboarding procedure
----------------------------------------------------
PlanExecutionTask
----------------------------------------------------
2022-05-25T06:02:37.382Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Starting PlanExecutionTask. Plan link: /relocation/onboarding/plan/15b6c88f-00eb-41f2-9bf8-631de817f95b, execution ID: 21a6c776-34c1-44b1-87cb-e67e91d202a2
2022-05-25T06:02:37.389Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-14' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Updating plan status.
2022-05-25T06:02:37.391Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Old status: OK, new status: EXECUTING.
2022-05-25T06:02:37.489Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Querying for first batch of deployments.
2022-05-25T06:02:37.497Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Getting batch.
2022-05-25T06:02:37.500Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Executing batch.
2022-05-25T06:02:37.508Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Waiting for batch to finish.
2022-05-25T06:02:38.514Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-14' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Waiting for batch to finish.
2022-05-25T06:02:39.523Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Updating plan status.
2022-05-25T06:02:39.526Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Old status: EXECUTING, new status: OK.
2022-05-25T06:02:39.536Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/28968af2-0162-4e10-b98e-1837d1218624] Finished.
----------------------------------------------------
DeploymentExecutionTask
** This task would begin between Executing batch and Updating Plan status of PlanExecutionTask. During this phase deployment get's onboarded **
----------------------------------------------------
2022-05-25T06:02:37.502Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Starting DeploymentExecutionTask. Deployment link: /relocation/onboarding/deployment/cd312a40-dc55-43a0-ad07-3c6d2efada67, execution ID: 21a6c776-34c1-44b1-87cb-e67e91d202a2
2022-05-25T06:02:37.511Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Identifying eligibility to onboard the deployment.
2022-05-25T06:02:37.514Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Deployment /relocation/onboarding/deployment/cd312a40-dc55-43a0-ad07-3c6d2efada67 eligible to onboard : true
2022-05-25T06:02:37.516Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Initializing DeploymentExecutionTask.
2022-05-25T06:02:37.523Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Updating resource placements for project /provisioning/resources/projects/e14d678c-f1c1-4a0d-9359-7c2aadbb3736
2022-05-25T06:02:37.528Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating blueprint.
2022-05-25T06:02:37.532Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating deployment.
2022-05-25T06:02:37.615Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating networks.
2022-05-25T06:02:38.142Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating machines.
2022-05-25T06:02:38.574Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Allocating IP addresses.
2022-05-25T06:02:38.602Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating load balancers.
2022-05-25T06:02:38.608Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating disks.
2022-05-25T06:02:38.616Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating securityGroups.
2022-05-25T06:02:38.620Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Creating network gateways.
2022-05-25T06:02:38.624Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Notify deployment onboarded.
2022-05-25T06:02:38.686Z INFO relocation [host='relocation-service-app-6b4db4f547-wnt5p' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.w.e.DeploymentExecutionTask - [8980/relocation/api/wo/execute-deployment/996518ae-e277-47b4-9cb5-95316945ded9] Finished.Investigation
Onboarding feature logs it's information under relocation-service logs
After we create a deployment plan, when we execute to onboard as discussed before there is a PlanExecutionTask which is triggered
Each PlanExecutionTask is associated with an execution id
2022-05-10T01:51:22.198Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-10' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/cd323-4e6b-4c0d-b972-3abc123cba] Starting PlanExecutionTask. Plan link: /relocation/onboarding/plan/5a3d36ba-5df9-4g8c-a665-c7sh3k4kff, execution ID: 72dhrd45-3aa2-4ff9-9941-a4fjhfjfjfja3c What we saw from the logs was that there were multiple executions but the last one or the attempt failed with an exception
2022-05-10T02:03:40.394Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-16' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Starting PlanExecutionTask. Plan link: /relocation/onboarding/plan/4b427637-6121-49cc-a1cc-5fe2e4581ea5, execution ID: bef9fc06-1ecf-4c21-a04f-bc53ed881f48
2022-05-10T02:03:40.403Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-9' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b69bd8e8-b641-4f9a-b44a-0b022ae5055e] Updating plan status.
2022-05-10T02:03:40.404Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-8' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Old status: OK, new status: EXECUTING.
2022-05-10T02:03:40.484Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-13' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055ee] Updating plan status.
2022-05-10T02:03:40.485Z INFO relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-14' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Old status: EXECUTING, new status: ERROR.
2022-05-10T02:03:40.492Z ERROR relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-updates-21' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] java.lang.RuntimeException: Validation failed.
2022-05-10T02:03:40.493Z ERROR relocation [host='relocation-service-app-747649cd5-hrlbn' thread='xn-index-queries-12' user='' org='' trace=''] c.v.r.wo.execution.PlanExecutionTask - [8980/relocation/api/wo/execute-plan/b6fnfnnr8-b641-4f9a-bcca-0bfjrjtjd055e] Failed: RuntimeException: Validation failed.Searching with the deployment id which was deleted from tango-blueprint logs we found out that the resource under the deployment ( virtual machine) was onboard and then deleted
2022-05-10T02:03:10.264Z INFO tango-blueprint [host='tango-blueprint-service-app-c6b9f9d9f-4h9wp' thread='tasks-2' user='relocation-8NNrKCAYnu5oyMLf(configadmin)' org='***' project='67e0331a-70ea-47ed-b9f0-994eb965e552' deployment='67***b-4279-4***-b***-1*******d' tile='311b69a3-5378-4da2-bc32-a6b870ed305e' trace='0f9d6925-904f-4202-b8c9-dd90a8722fe2'] com.vmware.tango.blueprint.gateway.DeploymentGateway - Deployment resource Cloud.vSphere.Machine successful. DeploymentId:67***b-4279-4***-b***-1*******d, ResourceId:f72b***-f***-3***-9***-2*****1
*
*
*
2022-05-10T02:07:09.188Z INFO tango-blueprint [host='tango-blueprint-service-app-c6b9f9d9f-kdxqc' thread='tasks-5' user='configadmin' org='***' project='67e0331a-70ea-47ed-b9f0-994eb965e552' deployment='67***b-4279-4***-b***-1*******d' request='0cc4024b-bb26-4d3c-a37c-82e31512a2fd' flow='9ff5dc4e-d6f6-4d7a-a826-8390603dc34a' task='f13077fc-103f-4f38-a89c-e60971fc96b0' tile='a5322a42-0cdd-4f9c-86d1-4e3748dea2ac' trace='b6a259b0-9816-4b0d-82fc-822af5a287c8'] com.vmware.tango.blueprint.gateway.DeploymentGateway - Deployment resource deleted successfully. DeploymentId:67***b-4279-4***-b***-1*******d, ResourceId:f72b***-f***-3***-9***-2*****1
Here comes the question , why would someone delete a deployment once it was onboarded
You would find the answer above , when the onboarding execution failed , few of the resources which were selected for this process were partially onboarded
Now since every resource is part of same deployment , if one deletes the deployment , all the underlying resources would be deleted.
This is the reason hundreds of virtual machines were deleted
How do you find out who deleted the deployment is from the below snippet under catalog-service-app logs
2022-05-10T02:04:36.340Z INFO catalog-service-app [host='cata
log-service-app-***-***' thread='http-nio-8000-exec-18' user='configadmin' org='*****' trace='b6a259b0-9816-4b0d-82fc-822af5a287c8'] c.v.t.d.s.ResourceActionRequestService - Submitted action Delete for
deployment 67825eab-4279-4ccd-b100-1fe929623a9d resource nullThe user configadmin is the one who submitted the action called delete
For example , if i submit delete in my lab for a deployment then it would be shown as below
*** user here is catalog-xxxxxx* which is a system/service user. Which states that this resource was deleted by vRA as it was expired ***
2022-05-27T03:06:21.115Z INFO catalog-service-app [host='catalog-service-app-75f5666cf-78ncp' thread='http-nio-8000-exec-19' user='catalog-VclHatUt32e13g3w' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='3b0cc70f-8852-475b-975a-43a962d2e7c2'] c.v.t.d.s.ResourceActionRequestService - Submitted action Delete for deployment 59befaa7-876d-480f-9bca-1c64f1c83e87 resource null
*** If it was a user initiated deletion , like me triggering one then it would be like below ***
2022-05-26T03:59:56.591Z INFO catalog-service-app [host='catalog-service-app-75f5666cf-78ncp' thread='http-nio-8000-exec-19' user='arun' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='a26fc3d9-7824-41b2-afee-d8d08bf6fd36'] c.v.t.d.s.ResourceActionRequestService - Submitted action Delete for deployment 6eb088c3-d9e6-4580-8e27-d924e93f6681 resource null
Note: configadmin is a local account which has privileges of highest level. It's recommended to use individual accounts with certain roles assigned for proper governance
There is an enhancement which is being made in the product which would inform user how many resources ( virtual machine , network , disks ...etc... ) would be impacted when he / she deletes the deployment.
This will give them a clue for not to go and directly submit a deletion . This should be out in the next release of vRA 8.8.1
Commentaires