Problem Statement
On a greenfield installation of vRA 8.7 which is integrated with SaltStack Config , we have an option to choose "Running Environment"
This screenshot below explains how a default SaltStack config deployed through vRSLCM
is presented under infrastructure tab
When you enter the password for root , without selecting any option under "Running Environment" , it successfully validates
But, the moment you select "Running Environment" we see an exception where the validation fails
before we see the exception , there is an abx integration run which occurs which gives you more details about the exception
Running in polyglot!
[2022-04-06 13:56:22,183] [INFO] - [saltstack-integration] Validating Salt Stack Config Server credentials...
[2022-04-06 13:56:22,183] [INFO] - [saltstack-integration] Authenticating to a Salt Stack Config Server with url [https://ss.cap.org//account/login]...
[2022-04-06 13:56:22,184] [INFO] - [saltstack-integration] Retrieving credentials from auth credentials link at [/core/auth/credentials/f0c26468-4c1b-4a62-b33a-b04d7c03390e]...
[2022-04-06 13:56:22,304] [INFO] - [saltstack-integration] Successfully retrieved credentials from auth credentials link
[2022-04-06 13:56:22,304] [INFO] - [saltstack-integration] Retrieving Salt Stack Config Server XSRF token from url [https://ss.cap.org//account/login]...
/run/abx-polyglot/function/urllib3/connectionpool.py:1050: InsecureRequestWarning: Unverified HTTPS request is being made to host 'ss.cap.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
[2022-04-06 13:56:22,327] [ERROR] - [saltstack-integration] Failed to validate Salt Stack Config Server credentials: Failed to authenticate to a Salt Stack Config Server: Failed to retrieve Salt Stack Config Server XSRF token: 403 Client Error: Forbidden for url: https://ss.cap.org//account/login
Finished running action code.
Exiting python process.
Python process exited.
Max Memory Used: 22 MB
The reason for the exception is as below
Failed to validate Salt Stack Config Server credentials: Failed to authenticate to a Salt Stack Config Server: Failed to retrieve Salt Stack Config Server XSRF token: 403 Client Error: Forbidden for url: https://ss.cap.org//account/login
exception under provisioning-service-app.log
2022-04-06T16:37:34.118Z WARN provisioning [host='provisioning-service-app-6885766867-kgk4l' thread='reactor-http-epoll-10' user='provisioning-RVgAJFw9LrOYkeUr(arun)' org='c2eae67a-ff6d-4dae-9fd3-6594352a1f8a' trace='dc45aa9b-4b4e-47d3-8176-8321b1a2336a' parent='4a10178e-bf5f-48d0-8928-ae7a84e3aff4' span='d1977a94-1448-45a5-b93a-e449c8a76b60'] c.v.xenon.common.ServiceErrorResponse.create:85 - message: Failed to authenticate, please check your credentials or if the host is reachable, statusCode: 400, serverErrorId: 9c245260-075a-4dc0-bbe2-fb13b0e5d0bd: Caused by java.lang.RuntimeException: Failed to authenticate, please check your credentials or if the host is reachable
at com.vmware.xenon.common.SpringHostUtils.responseEntityToOperation(SpringHostUtils.java:952)
at com.vmware.xenon.common.SpringHostUtils.lambda$sendRequest$4(SpringHostUtils.java:289)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at reactor.core.publisher.MonoToCompletableFuture.onNext(MonoToCompletableFuture.java:64)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)
at io.opentracing.contrib.reactor.TracedSubscriber.lambda$onNext$2(TracedSubscriber.java:69)
at io.opentracing.contrib.reactor.TracedSubscriber.withActiveSpan(TracedSubscriber.java:95)
at io.opentracing.contrib.reactor.TracedSubscriber.onNext(TracedSubscriber.java:69)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
at io.opentracing.contrib.reactor.TracedSubscriber.lambda$onNext$2(TracedSubscriber.java:69)
at io.opentracing.contrib.reactor.TracedSubscriber.withActiveSpan(TracedSubscriber.java:95)
at io.opentracing.contrib.reactor.TracedSubscriber.onNext(TracedSubscriber.java:69)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)
*
*
*
*
*
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)Remediation
Method 1 (Greenfeild Scenario )
If you have a SaltStack which was recently deployed and doesn't have any resources mapped to this integration , then simply delete the integration and recreate it
Before
After
See the difference , ensure when your adding the hostname in the integration
When vRSLCM add SaltStack integration , it uses URL ( https://<<saltstackhostname>>/ ) which is when you would see the problem
Once you remove the integration and then add it back again with just FQDN and not the URL of the SaltStack server , then we select an Running environment , it all works fine.
Method 2 ( Brownfield Scenario )
When you have resources being managed by SaltStack
Integration information is stored inside provisioning-db of vRealize Automation environment
To login into database use
vracli dev psqlAccept the warning that it's a developer command and ensure you know what you are changing
Below is the screenshot and output of the table where the integration information is stored.
The table is known as endpoint_state , this is present inside provisioning-db
To connect to provisioning-db use the below command
\c provisioning-db
root@vra [ ~ ]# vracli dev psql
This execution will be recorded!
'psql' is a developer command. Type 'yes' if you want to continue, or 'no' to stop: yes
2022-04-06 14:14:43,439 [INFO] Logging into database template1
psql (10.18)
Type "help" for help.
template1=# \c provisioning-db
You are now connected to database "provisioning-db" as user "postgres".
provisioning-db=# \x
Expanded display is on.
provisioning-db=# select * from endpoint_state where name = 'vssc_idm';
-[ RECORD 1 ]-------------------+---------------------------------------------------------------------------------------------------------------------
document_self_link | /resources/endpoints/b2b02510-b0d5-46cf-9248-570b3d1bd58d
document_auth_principal_link | /provisioning/auth/csp/users/cgs-lecvl28lpzqwhozt@provisioning-client.local
document_expiration_time_micros | 0
document_owner |
document_update_action | PATCH
document_update_time_micros | 1646141393852000
document_version | 1
id | 6c2679af-a23c-4c88-8af0-3380305e3cde
name | vssc_idm
c_desc |
custom_properties | {"hostName": "https://ss.cap.org/", "isExternal": "true", "privateKeyId": "root"}
tenant_links | ["/tenants/organization/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a", "/tenants/project/1f32c781c7bac475-7f703c5265a63d87"]
group_links |
tag_links |
org_auth_link | /tenants/organization/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a
project_auth_link |
owner_auth_link |
msp_auth_link |
creation_time_micros |
region_id |
endpoint_links |
compute_host_link | /resources/compute/c6f3a8ac-c700-41b2-a91d-91a3fdd73765
expanded_tags |
document_creation_time_micros | 1646141393802000
endpoint_type | saltstack
auth_credentials_link | /core/auth/credentials/eeb389af-fcd6-4b06-a0e9-5d178f128eed
compute_link | /resources/compute/c6f3a8ac-c700-41b2-a91d-91a3fdd73765
compute_description_link | /resources/compute-descriptions/8a689d42-24f7-4f6d-b362-e85b6dc6f423
resource_pool_link | /resources/pools/1f32c781c7bac475-7f703c5265a63d87
parent_link |
associated_endpoint_links |
endpoint_properties | {"hostName": "https://ss.cap.org/", "privateKeyId": "root"}
maintenance_mode |
mobility_endpoint_links |
provisioning-db=#Look at the custom_properties section , this is how it is out of the box
custom_properties | {"hostName": "https://ss.cap.org/", "isExternal": "true", "privateKeyId": "root"}We would add an additional property called dcID and change the hostname to FQDN than a URL and keep endpointId blank.
update endpoint_state set custom_properties = '{"dcId": "onprem", "hostName": "ss.cap.org", "endpointId": "", "isExternal": "true", "privateKeyId": "root"}' where name = 'vssc_idm';Along with it , we would have to change endpoint_properties too. This has to reflect FQDN than the whole url
endpoint_properties | {"hostName": "https://ss.cap.org/", "privateKeyId": "root"}Note : Before making changes i'll take a snapshot of vRA appliance
As we already logged into the database before , let's go ahead and make the change. Execute below query and ensure its successful
update endpoint_state set custom_properties = '{"dcId": "onprem", "hostName": "ss.cap.org", "endpointId": "", "isExternal": "true", "privateKeyId": "root"}' where name = 'vssc_idm';
update endpoint_state set endpoint_properties = '{"hostName": "ss.cap.org", "privateKeyId": "root"}' where name = 'vssc_idm';
provisioning-db=# update endpoint_state set custom_properties = '{"dcId": "onprem", "hostName": "ss.cap.org", "endpointId": "", "isExternal": "true", "privateKeyId": "root"}' where name = 'vssc_idm';
UPDATE 1
provisioning-db=# update endpoint_state set endpoint_properties = ' {"hostName": "ss.cap.org", "privateKeyId": "root"}' where name = 'vssc_idm';
UPDATE 1
provisioning-db=# select * from endpoint_state where name = 'vssc_idm';
-[ RECORD 1 ]-------------------+---------------------------------------------------------------------------------------------------------------------
document_self_link | /resources/endpoints/b2b02510-b0d5-46cf-9248-570b3d1bd58d
document_auth_principal_link | /provisioning/auth/csp/users/cgs-lecvl28lpzqwhozt@provisioning-client.local
document_expiration_time_micros | 0
document_owner |
document_update_action | PATCH
document_update_time_micros | 1646141393852000
document_version | 1
id | 6c2679af-a23c-4c88-8af0-3380305e3cde
name | vssc_idm
c_desc |
custom_properties | {"dcId": "onprem", "hostName": "ss.cap.org", "endpointId": "", "isExternal": "true", "privateKeyId": "root"}
tenant_links | ["/tenants/organization/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a", "/tenants/project/1f32c781c7bac475-7f703c5265a63d87"]
group_links |
tag_links |
org_auth_link | /tenants/organization/c2eae67a-ff6d-4dae-9fd3-6594352a1f8a
project_auth_link |
owner_auth_link |
msp_auth_link |
creation_time_micros |
region_id |
endpoint_links |
compute_host_link | /resources/compute/c6f3a8ac-c700-41b2-a91d-91a3fdd73765
expanded_tags |
document_creation_time_micros | 1646141393802000
endpoint_type | saltstack
auth_credentials_link | /core/auth/credentials/eeb389af-fcd6-4b06-a0e9-5d178f128eed
compute_link | /resources/compute/c6f3a8ac-c700-41b2-a91d-91a3fdd73765
compute_description_link | /resources/compute-descriptions/8a689d42-24f7-4f6d-b362-e85b6dc6f423
resource_pool_link | /resources/pools/1f32c781c7bac475-7f703c5265a63d87
parent_link |
associated_endpoint_links |
endpoint_properties | {"hostName": "ss.cap.org", "privateKeyId": "root"}
maintenance_mode |
mobility_endpoint_links |
As one can see from the above update , we did change the custom_properties of the SSC integration in vRA
Exit the database by executing
\qNow let's reboot saltstack , log out of vRA and log back in again .
See if the FQDN is back in the hostname rather than the URL. If that's the case it would successfully authenticate with the "Running Environment " in place
Running in polyglot!
[2022-04-06 17:13:36,475] [INFO] - [saltstack-integration] Validating Salt Stack Config Server credentials...
[2022-04-06 17:13:36,475] [INFO] - [saltstack-integration] Authenticating to a Salt Stack Config Server with url [https://ss.cap.org/account/login]...
[2022-04-06 17:13:36,475] [INFO] - [saltstack-integration] Retrieving credentials from auth credentials link at [/core/auth/credentials/d7ea970e-cdca-42bc-b53d-ddac713a8666]...
[2022-04-06 17:13:36,519] [INFO] - [saltstack-integration] Successfully retrieved credentials from auth credentials link
[2022-04-06 17:13:36,519] [INFO] - [saltstack-integration] Retrieving Salt Stack Config Server XSRF token from url [https://ss.cap.org/account/login]...
/run/abx-polyglot/function/urllib3/connectionpool.py:1050: InsecureRequestWarning: Unverified HTTPS request is being made to host 'ss.cap.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
[2022-04-06 17:13:36,544] [INFO] - [saltstack-integration] Successfully retrieved Salt Stack Config Server XSRF token
/run/abx-polyglot/function/urllib3/connectionpool.py:1050: InsecureRequestWarning: Unverified HTTPS request is being made to host 'ss.cap.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
[2022-04-06 17:13:36,633] [INFO] - [saltstack-integration] Successfully authenticated to a Salt Stack Config Server
[2022-04-06 17:13:36,634] [INFO] - [saltstack-integration] Successfully validated Salt Stack Config Server credentials
Finished running action code.
Exiting python process.
Python process exited.
Max Memory Used: 21 MBIn Short
Issue is seen due to the fact there is a URL rather than FQDN and when it's trying to execute an API to authentication it get's a 403 error
Unless we fix this issue you will not be able to successfully validate running environment
If it's a new environment with no SaltStack resources , go ahead and delete the integration and re-create it
If it's an existing integration with resources in place , then modify the database as shown above
1. connect to postgres database
vracli dev psql
2. connect to provisiioning-db
\c provisioning-db
3. enable expanded display
\x
4. Update custom_properties value where the hostname is set to URL of SaltStack node than an FQDN. Remember to change in endpoint_state table as shown below. Sometimes the name of the integration might be different if it's changed from UI. So change it accordingly.
update endpoint_state set custom_properties = '{"dcId": "onprem", "hostName": "FQDN-SALTSTACKNODE", "endpointId": "", "isExternal": "true", "privateKeyId": "root"}' where name = 'vssc_idm';
5. Update endpoint_properties column value where you have hostname set to URL to FQDN. Almost same as above
provisioning-db=# update endpoint_state set endpoint_properties = ' {"hostName": "FQDN-SALTSTACKNODE", "privateKeyId": "root"}' where name = 'vssc_idm';
Now add "Running Environment" and then validate. You should see a successful validation in place
Comments