We've recently seen issues while trying update resource elements in vRealize Orchestrator 7.x
Some of the exceptions you would see when the vRO workflow is executed is as below. Below excerpts are from scripting.log
2021-03-09 11:42:35.070+0000 [WorkflowExecutorPool-Thread-2] ERROR {|__SYSTEM|administrator@vsphere.local:updateResourceEleme:f896e1b8-4b5e-484e-b5d1-db04390a1bcc:token=e98c4043-52d4-4375-96b4-c699da88f34d} [ResourceElementManagementServiceImpl] Failed to commit 'Resource saved.'. Reason: null
java.lang.NullPointerException
Cannot set content from mime attachment : ch.dunes.util.DunesServerException: org.springframework.orm.jpa.JpaSystemException: Transaction was marked for rollback only; cannot commit; nested exception is org.hibernate.TransactionException: Transaction was marked for rollback only; cannot commit
One of the main reasons why this is not working is because the jgit garbage collection is not working as expected in the background
2021-03-09 11:42:23.354+0000 [http-nio-127.0.0.1-8280-exec-6] ERROR {} [ContentVersionManagementServiceImpl] Failed to disable auto push: org.eclipse.jgit.api.errors.TransportException: /storage/db/vcodata/git/__SYSTEM.git: internal server error
java.lang.RuntimeException: org.eclipse.jgit.api.errors.TransportException: /storage/db/vcodata/git/__SYSTEM.git: internal server error
at com.vmware.o11n.git.StagingRepositoryService.checkout(StagingRepositoryService.java:226)
at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl$ReusableStaginRepository.<init>(ContentVersionManagementServiceImpl.java:495)
at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl$StagingRepositoryPooledObjectFactory.create(ContentVersionManagementServiceImpl.java:455)
at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl$StagingRepositoryPooledObjectFactory.create(ContentVersionManagementServiceImpl.java:445)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:58)
2021-03-10 03:00:00.000+0000 [vcoSystemTaskScheduler-3] DEBUG {} [ContentVersionManagementServiceImpl] Starting garbage collection in all repositories.
2021-03-10 03:00:05.931+0000 [vcoSystemTaskScheduler-3] INFO {} [BareRepositoryService] Started garbage collection for __SYSTEM.git
2021-03-10 03:00:06.352+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService] Failed to run garbage collection for __SYSTEM.git. Reason:
2021-03-10 03:00:06.352+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService] Garbage collection failed.
org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed.
at org.eclipse.jgit.api.GarbageCollectCommand.call(GarbageCollectCommand.java:229)
at com.vmware.o11n.git.BareRepositoryService.runGc(BareRepositoryService.java:109)
at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl.lambda$compact$3(ContentVersionManagementServiceImpl.java:260)
at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl.wrapInTransaction(ContentVersionManagementServiceImpl.java:111)
at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl.compact(ContentVersionManagementServiceImpl.java:256)
at com.vmware.o11n.service.version.ContentVersionManagementServiceImpl.lambda$compactAll$2(ContentVersionManagementServiceImpl.java:251)
at java.util.ArrayList.forEach(ArrayList.java:1257)
Garbage Collection runs everyday at 3 AM. If you notice garbage collection failures as shown below then execute the steps mentioned in the remediation section
2021-03-06 03:00:12.167+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService] Garbage collection failed.
org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed.
2021-03-07 03:00:00.545+0000 [vcoSystemTaskScheduler-4] ERROR {} [BareRepositoryService] Garbage collection failed.
org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed.
2021-03-08 03:00:03.969+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService] Garbage collection failed.
org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed.
2021-03-09 03:00:00.469+0000 [vcoSystemTaskScheduler-4] ERROR {} [BareRepositoryService] Garbage collection failed.
org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed.
2021-03-10 03:00:06.352+0000 [vcoSystemTaskScheduler-3] ERROR {} [BareRepositoryService] Garbage collection failed.
org.eclipse.jgit.api.errors.JGitInternalException: Garbage collection failed.
To resolve this issue follow these steps
Download JGit cli client , below command would work if internet is available, else execute this command on node which has internet access and copy the file into vRA node
curl -k --output jgit https://repo.eclipse.org/content/groups/releases//org/eclipse/jgit/org.eclipse.jgit.pgm/5.4.0.201906121030-r/org.eclipse.jgit.pgm-5.4.0.201906121030-r.sh
2. Move jgit to /usr/bin, set it to executable and change its owner to vco and make it executable
mv jgit /usr/bin/
chown vco:vco /usr/bin/jgit
chmod +x /usr/bin/jgit
3. Remove the HEAD ref of the MASTER
mv /storage/db/vcodata/git/__SYSTEM.git/refs/heads/master /var/lib/vco/master_ref.backup
4. Check where the HEAD should be.Get the commit ref sha from the first line. ex: 01bb5f58716dc12016b4fd8798c7fa8c91c76bf3
cd /storage/db/vcodata/git/__SYSTEM.git
jgit log
5. Login as vco
su - vco
6. Create the master ref
echo '01bb5f58716dc12016b4fd8798c7fa8c91c76bf3' > /storage/db/vcodata/git/__SYSTEM.git/refs/heads/master
7. Run jgit garbage collection
cd /storage/db/vcodata/git/__SYSTEM.git
jgit gc
8. Login as root again and make sure everything in the git folder is owned by vco:vco
exit
cd /storage/db/vcodata/git/__SYSTEM.git
chown -R vco:vco
9. Restart the vRO server
service vco-server restart
This should fix any garbage collection issues
Comments