Unable to update resource elements in vRealize Orchestrator

We've recently seen issues while trying update resource elements in vRealize Orchestrator 7.x

Some of the exceptions you would see when the vRO workflow is executed is as below. Below excerpts are from scripting.log

2021-03-09 11:42:35.070+0000 [WorkflowExecutorPool-Thread-2] ERROR {|__SYSTEM|administrator@vsphere.local:updateResourceEleme:f896e1b8-4b5e-484e-b5d1-db04390a1bcc:token=e98c4043-52d4-4375-96b4-c699da88f34d} [ResourceElementManagementServiceImpl] Failed to commit 'Resource saved.'. Reason: null 
java.lang.NullPointerException 

Cannot set content from mime attachment : ch.dunes.util.DunesServerException: org.springframework.orm.jpa.JpaSystemException: Transaction was marked for rollback only; cannot commit; nested exception is org.hibernate.TransactionException: Transaction was marked for rollback only; cannot commit 

One of the main reasons why this is not working is because the jgit garbage collection is not working as expected in the background

2021-03-09 11:42:23.354+0000 [http-nio-127.0.0.1-8280-exec-6] ERROR {} [ContentVersionManagementServiceImpl] Failed to disable auto push: org.eclipse.jgit.api.errors.TransportException: /storage/db/vcodata/git/__SYSTEM.git: internal server error 
java.lang.RuntimeException: org.eclipse.jgit.api.errors.TransportException: /storage/db/vcodata/git/__SYSTEM.git: internal server error 
        at com.vmware.o11n.git.StagingRepositoryService.checkout(StagingRepositoryService.java:226) 
        at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl$ReusableStaginRepository.<init>(ContentVersionManagementServiceImpl.java:495) 
        at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl$StagingRepositoryPooledObjectFactory.create(ContentVersionManagementServiceImpl.java:455) 
        at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl$StagingRepositoryPooledObjectFactory.create(ContentVersionManagementServiceImpl.java:445) 
        at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:58) 


2021-03-10 03:00:00.000+0000 [vcoSystemTaskScheduler-3] DEBUG {} [ContentVersionManagementServiceImpl] Starting garbage collection in all repositories. 
2021-03-10 03:00:05.931+0000 [vcoSystemTaskScheduler-3] INFO  {} [BareRepositoryService] Started garbage collection for __SYSTEM.git 
2021-03-10 03:00:06.352+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService] Failed to run garbage collection for __SYSTEM.git. Reason: 
2021-03-10 03:00:06.352+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService]         Garbage collection failed. 
org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed. 
        at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:229) 
        at com.vmware.o11n.git.BareRepositoryService.runGc(BareRepositoryService.java:109) 
        at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl.lambda$compact$3(ContentVersionManagementServiceImpl.java:260) 
        at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl.wrapInTransaction(ContentVersionManagementServiceImpl.java:111) 
        at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl.compact(ContentVersionManagementServiceImpl.java:256) 
        at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl.lambda$compactAll$2(ContentVersionManagementServiceImpl.java:251) 
        at java.util.ArrayList.forEach(ArrayList.java:1257)

Garbage Collection runs everyday at 3 AM. If you notice garbage collection failures as shown below then execute the steps mentioned in the remediation section

2021-03-06 03:00:12.167+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService]         Garbage collection failed.
org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed.
2021-03-07 03:00:00.545+0000 [vcoSystemTaskScheduler-4] ERROR {} [BareRepositoryService]         Garbage collection failed.
org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed.
2021-03-08 03:00:03.969+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService]         Garbage collection failed.
org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed.
2021-03-09 03:00:00.469+0000 [vcoSystemTaskScheduler-4] ERROR {} [BareRepositoryService]         Garbage collection failed.
org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed.
2021-03-10 03:00:06.352+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService]         Garbage collection failed.
org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed.

To resolve this issue follow these steps

1. Download JGit cli client , below command would work if internet is available, else execute this command on node which has internet access and copy the file into vRA node

curl -k --output jgit https://repo.eclipse.org/content/groups/releases//org/eclipse/jgit/org.eclipse.jgit.pgm/5.4.0.201906121030-r/org.eclipse.jgit.pgm-5.4.0.201906121030-r.sh

2. Move jgit to /usr/bin, set it to executable and change its owner to vco and make it executable

mv jgit /usr/bin/
chown vco:vco /usr/bin/jgit
chmod +x /usr/bin/jgit

3. Remove the HEAD ref of the MASTER

mv /storage/db/vcodata/git/__SYSTEM.git/refs/heads/master /var/lib/vco/master_ref.backup

4. Check where the HEAD should be.Get the commit ref sha from the first line. ex: 01bb5f58716dc12016b4fd8798c7fa8c91c76bf3

cd /storage/db/vcodata/git/__SYSTEM.git
jgit log

5. Login as vco

su - vco

6. Create the master ref

echo '01bb5f58716dc12016b4fd8798c7fa8c91c76bf3' > /storage/db/vcodata/git/__SYSTEM.git/refs/heads/master

7. Run jgit garbage collection

cd /storage/db/vcodata/git/__SYSTEM.git
jgit gc

8. Login as root again and make sure everything in the git folder is owned by vco:vco

exit
cd /storage/db/vcodata/git/__SYSTEM.git
chown -R vco:vco

9. Restart the vRO server

service vco-server restart

This should fix any garbage collection issues