Operational Defect Database
BugZero updated this defect 46 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
VMware | 96304
NSX Edge syslog reports "Failed to collect metrics, Exception:badly formed hexadecimal UUID string" while doing the audit_log_health.remote_logging_server_error alarm check for all the rsyslog exporters.
Last update date:
4/4/2024
Affected products:
NSX-T
Affected releases:
3.x
Fixed releases:
No fixed releases provided.
Description:
Symptoms
You will see the below log in NSX Edge's syslog. 2023-10-31T13:00:34.914Z NSX 3186 - [nsx@6876 comp="nsx-edge" subcomp="nsx-sha" username="root" level="WARNING" s2comp="metric-collector"] Failed to collect metrics, Exception:badly formed hexadecimal UUID string, trace:Traceback (most recent call last):#012 File "/opt/vmware/nsx-netopa/lib/python/sha/core/collector/_metric_collector.py", line 157, in _collect#012 data = metric_usage()#012 File "/opt/vmware/nsx-netopa/lib/python/sha/contrib/metric/utils/_time_out_cache.py", line 15, in wrapper#012 record[1] = fn(*args, **kwargs)#012 File "/opt/vmware/nsx-netopa/lib/python/sha/contrib/metric/audit_log_health_for_remote_error_status.py", line 93, in usage#012 key = uuid.UUID(cur_logging_server[_EXPORTER_NAME])#012 File "/opt/vmware/nsx-netopa/libexec/python-3.8.3/lib64/python3.8/uuid.py", line 169, in __init__#012 raise ValueError('badly formed hexadecimal UUID string')#012ValueError: badly formed hexadecimal UUID string Lab replication and findings :**************************************NSX version : 3.2.1.2.0.205412191. CLI configuration :edge01> get logging-serversThu Nov 09 2023 UTC 06:23:27.083You create a logging-server using Edge CLI.edge01> set logging-server 1.1.1.1:514 proto tcp level warn WARNING - You are configuring tcp-based log forwarding. This will send sensitive information unencrypted over the network. The Splunk App for NSX-T only accepts TLS connections.edge01> get logging-serversThu Nov 09 2023 UTC 06:24:28.8851.1.1.1:514 proto tcp level warning exporter_name 3b2439a2-58c8-40f5-9fb3-2a456db44b53Logging server 1.1.1.1 gets configured successfully with an exporter name as UUID string 3b2439a2-58c8-40f5-9fb3-2a456db44b53.From /etc/rsyslog.conf file :-$ActionQueueType LinkedList # nsx exporter: 3b2439a2-58c8-40f5-9fb3-2a456db44b53*.warning @@1.1.1.1:514;RFC5424fmt # nsx exporter: 3b2439a2-58c8-40f5-9fb3-2a456db44b53GET : https://192.168.120.1/api/v1/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters{ "_schema": "NodeSyslogExporterPropertiesListResult", "_self": { "href": "/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters", "rel": "self" }, "result_count": 3, "results": [ { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/6bd7551b-74b8-4218-897b-ad7c78f092f7", "rel": "self" }, "exporter_name": "6bd7551b-74b8-4218-897b-ad7c78f092f7", "level": "WARNING", "port": 514, "protocol": "TCP", "server": "2.2.2.2" }, { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/3b2439a2-58c8-40f5-9fb3-2a456db44b53", "rel": "self" }, "exporter_name": "3b2439a2-58c8-40f5-9fb3-2a456db44b53", "level": "WARNING", "port": 514, "protocol": "TCP", "server": "1.1.1.1" }, { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/925eb71f-1040-48de-b66f-78e5f5033d24", "rel": "self" }, "exporter_name": "925eb71f-1040-48de-b66f-78e5f5033d24", "level": "INFO", "port": 514, "protocol": "UDP", "server": "192.168.120.200" } ]}2. API configuration :POST : https://192.168.120.1/api/v1/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters{ "exporter_name": "Amit-exporter", "facilities": ["KERN", "USER"], "level": "INFO", "msgids": ["tcpin", "tcpout"], "port": 514, "protocol": "TCP", "server": "5.5.5.5"}GET : https://192.168.120.1/api/v1/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters{ "_schema": "NodeSyslogExporterPropertiesListResult", "_self": { "href": "/transport-nodes/4aa82af6-7c8e-11ee-aed5-0050569dcbac/node/services/syslog/exporters", "rel": "self" }, "result_count": 4, "results": [ { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/6bd7551b-74b8-4218-897b-ad7c78f092f7", "rel": "self" }, "exporter_name": "6bd7551b-74b8-4218-897b-ad7c78f092f7", "level": "WARNING", "port": 514, "protocol": "TCP", "server": "2.2.2.2" }, { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/7e5a3cbb-6f06-44de-b90a-a4d29e04b2bd", "rel": "self" }, "exporter_name": "7e5a3cbb-6f06-44de-b90a-a4d29e04b2bd", "level": "WARNING", "port": 514, "protocol": "TCP", "server": "1.1.1.1" }, { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/925eb71f-1040-48de-b66f-78e5f5033d24", "rel": "self" }, "exporter_name": "925eb71f-1040-48de-b66f-78e5f5033d24", "level": "INFO", "port": 514, "protocol": "UDP", "server": "192.168.120.200" }, { "_schema": "NodeSyslogExporterProperties", "_self": { "href": "/node/services/syslog/exporters/Amit-exporter", "rel": "self" }, "exporter_name": "Amit-exporter", "facilities": [ "KERN", "USER" ], "level": "INFO", "msgids": [ "tcpin", "tcpout" ], "port": 514, "protocol": "TCP", "server": "5.5.5.5" } ]}From /etc/rsyslog.conf file :- $ActionQueueType LinkedList # nsx exporter: Amit-exporterif $msgid == 'tcpin' or $msgid == 'tcpout' then { # nsx exporter: Amit-exporterkern,user.info @@5.5.5.5:514;RFC5424fmt # nsx exporter: Amit-exporter} # nsx exporter: Amit-exporterroot@edge01:~#As we can see that if you configure exporter using API and exporter name as a string, then in /etc/rsyslog.conf, you will see nsx exporter name as string instead of UUID.
Cause
If an exporter is configured using API POST /node/services/syslog/exporters, a non-uuid string as exporter name gets created. When NSX does a audit_log_health.remote_logging_server_error check for all the rsyslog exporters, the events would be logged as it expects an UUID instead of a string as exporter name. UUID as an exporter name is only possible if exporter configuration is made with CLI instead of an API which generates a string as exporter name.
Impact / Risks
There is no impact.
Resolution
A mechanism for more robust handling of exporters is introduced in NSX 4.2.0 version.
Workaround
Use CLI to configure exporter.edge01> set logging-server 6.6.6.6:514 proto tcp level info WARNING - You are configuring tcp-based log forwarding. This will send sensitive information unencrypted over the network. The Splunk App for NSX-T only accepts TLS connections.NSX auto generates UUID when an exporter was configured using CLI.$ActionQueueType LinkedList # nsx exporter: 76738e49-93e2-4e88-9e6d-c6177e67cf05*.info @@6.6.6.6:514;RFC5424fmt # nsx exporter: 76738e49-93e2-4e88-9e6d-c6177e67cf05
