Error returned by axis2: Error occurred in transport

There was an interesting scenario where Linux deployments where stuck at a point where it's Installing OS ( this is not a standard template-based deployment but a kickstart deployment )

For more information on Linux Kickstart deployments from vRA click here

During this phase of Installing OS, we also install a Gugent ( Guest Agent for Linux )

This gugent installation was throwing exceptions as mentioned below

Linux Guest Agent Connection Log

/usr/share/gugent/axis2/logs/gugent-axis.log

[Wed May 13 22:15:04 2020] [debug] http_transport_sender.c(246) ctx_epr:https://<<IAAS-WEB-FQDN>>:443/VMPS2
[Wed May 13 22:15:04 2020] [debug] http_transport_sender.c(805) using axis2 native http sender.
[Wed May 13 22:15:04 2020] [...TRACE...] http_sender.c(252) Entry:axis2_http_sender_send
[Wed May 13 22:15:04 2020] [debug] http_sender.c(416) msg_ctx_id:urn:uuid:607582d6-9513-1ea1-30b8-005056889293
[Wed May 13 22:15:04 2020] [info]  [ssl client] Client certificate chain filenot specified
[Wed May 13 22:15:04 2020] [debug] ssl/ssl_utils.c(190) [ssl client] SSL certificate verified against peer
[Wed May 13 22:15:04 2020] [error] http_sender.c(3006) Authtype NTLM is notsupported

Linux Guest Agent Log

/usr/share/gugent/GuestAgent.log

2020-05-13 11:23:28 Application: [Information] Using the network enabled proxy ...
2020-05-13 11:23:28 Application: [Information] The vCAC endpoint is https://<<IAAS-WEB-FQDN>>:443/VMPS2.
2020-05-13 11:23:28 Application: [Information] The AXIS2C directory is axis2/.
2020-05-13 11:23:28 Application: [Information] Requesting work for agent ID 42088131-66e2-832d-1324-e5b8f05c5ee7.
2020-05-13 11:23:28 Application: [Information] Fetching a work item ...
2020-05-13 11:23:28 Application.Proxy: [Error]

2020-05-13 11:23:28 Application.Proxy: [Error] Error returned by axis2: Error occurred in transport
2020-05-13 11:23:28 Application.Proxy: [Error] This error code may indicate an incorrect endpoint address https://<<IAAS-WEB-FQDN>>:443/VMPS2, or a certificate issue.
2020-05-13 11:23:28 Application.Proxy: [Error] Consult axis2 log file at axis2/logs for more detail

As a first step, https://<<IAAS-WEB-FQDN>>:443/VMPS2 tried accessing the following URL. This URL was not returning a proper response.

VMPS is a VirtualMachineProvisionService which corresponds to Manager Service but not Web as being shown in the logs

This Guest Agent Workitem service configuration is actually stored under ManagerService.exe.Config

      <!-- Guest Agent WorkItem service. Called into by guest agents (gugents). Uses it's own payload-layer authentication. -->
      <service name="DynamicOps.VMPS.Service.GuestAgentWorkItemService" behaviorConfiguration="GuestAgentWorkItemServiceBehavior">
        <endpoint address="https://<<IAAS-MGR-FQDN>>:443/VMPS2" bindingConfiguration="GuestAgentWorkItemServiceBinding" binding="basicHttpBinding" contract="DynamicOps.VMPS.Contracts.IVMPSWorkItem" />
        <endpoint address="https://<<IAAS-MGR-FQDN>>:443/VMPS2/mex" binding="mexHttpsBinding" contract="IMetadataExchange" />
      </service>

https://IAAS-MGR-FQDN:443/VMPS2 returns following output

This is where we found out that when Guest Agent was being installed it was calling a wrong FQDN to fetch the workitem.

As explained above it was trying to reach out to IAAS-WEB-FQDN rather than trying to pull information from IAAS-MGR-FQDN

This misconfiguration was done on one of the repositories from where the guest is being downloaded.

Once the script which is installing guest agent is fixed with the right URL from where it has to pull the worktems or point it to the manager sevice fqdn , then issues with respect to deployments were resolved.