damhau/cucm-cdr

cucm-cdr's Issues

mapper_parsing_exception in version 7.10.1

Hello,

I realize that this GitHub Repo is 5 years old, but it is one of the few regarding CUCM CDRs with the ELK Stack. I appreciate the author’s work and contribution to the Community.

I am trying to set this up for ELK/Elastic Stack with release 7.10.1 of Kibana and the corresponding Elasticsearch and Logstash. Ideally, I want to get this all working inside Docker Containers. I have went through this setup, but I am receiving an error when I try to apply the Elasticsearch Template, the error I am seeing is the following:

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters: [cucm-cdr : {_source={enabled=true}, properties={destVideoTransportAddress_Port_Channel2={type=integer}, destVideoTransportAddress_IP_Channel2={type=integer}, origVideoCap_Codec={type=integer}, IncomingProtocolID={type=integer}, origSpan={type=integer}, destMediaCap_g723BitRate={type=integer}, destLegIdentifier={type=integer}, lastRedirectRedirectReason={type=integer}, dateTimeImport={type=date}, origMediaCap_maxFramesPerPacket={type=integer}, origMediaCap_g723BitRate={type=integer}, dateTimeConnect={type=integer}, joinOnBehalfOf={type=integer}, authorizationLevel={type=integer}, origVideoTransportAddress_Port={type=integer}, destConversationId={type=integer}, destMediaTransportAddress_Port={type=integer}, origVideoCap_Codec_Channel2={type=integer}, origVideoTransportAddress_Port_Channel2={type=integer}, destVideoChannel_Role_Channel2={type=integer}, lastRedirectRedirectOnBehalfOf={type=integer}, destSpan={type=integer}, destDTMFMethod={type=integer}, lastRedirectingRoutingReason={type=integer}, origMediaCap_Bandwidth={type=integer}, origCause_location={type=integer}, dateTimeDisconnect_formatted={type=date}, destVideoCap_Codec_Channel2={type=integer}, globalCallID_callId={type=integer}, origLegCallIdentifier={type=integer}, dateTimeOrigination={type=integer}, origRoutingReason={type=integer}, wasCallQueued={type=integer}, dateTimeConnect_formatted={type=date}, destVideoTransportAddress_Port={type=integer}, origVideoChannel_Role_Channel2={type=integer}, origCalledPartyRedirectOnBehalfOf={type=integer}, destVideoCap_Bandwidth_Channel2={type=integer}, destPrecedenceLevel={type=integer}, destVideoCap_Resolution={type=integer}, destMediaCap_Bandwidth={type=integer}, destVideoTransportAddress_IP={type=integer}, origVideoCap_Bandwidth_Channel2={type=integer}, cdrRecordType={type=integer}, calledPartyPatternUsage={type=integer}, origPrecedenceLevel={type=integer}, origVideoCap_Resolution_Channel2={type=integer}, origCause_value={type=integer}, globalCallID_callManagerId={type=integer}, dateTimeDisconnect={type=integer}, origMediaTransportAddress_IP={type=integer}, origMediaTransportAddress_Port={type=integer}, destMediaCap_payloadCapability={type=integer}, destVideoCap_Codec={type=integer}, duration={type=integer}, origVideoCap_Bandwidth={type=integer}, destCause_value={type=integer}, destMediaTransportAddress_IP={type=integer}, origDeviceName={index=not_analyzed, type=string}, destDeviceName={index=not_analyzed, type=string}, OutgoingProtocolID={type=integer}, origVideoTransportAddress_IP_Channel2={type=integer}, origNodeId={type=integer}, origCallTerminationOnBehalfOf={type=integer}, origIpAddr={type=integer}, destNodeId={type=integer}, origCalledPartyRedirectReason={type=integer}, origVideoCap_Resolution={type=integer}, destMediaCap_maxFramesPerPacket={type=integer}, destCallTerminationOnBehalfOf={type=integer}, destVideoCap_Bandwidth={type=integer}, origMediaCap_payloadCapability={type=integer}, callSecuredStatus={type=integer}, origVideoTransportAddress_IP={type=integer}, origConversationId={type=integer}, destVideoCap_Resolution_Channel2={type=integer}, destIpAddr={type=integer}, currentRoutingReason={type=integer}, totalWaitTimeInQueue={type=integer}}}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [cucm-cdr : {_source={enabled=true}, properties={destVideoTransportAddress_Port_Channel2={type=integer}, destVideoTransportAddress_IP_Channel2={type=integer}, origVideoCap_Codec={type=integer}, IncomingProtocolID={type=integer}, origSpan={type=integer}, destMediaCap_g723BitRate={type=integer}, destLegIdentifier={type=integer}, lastRedirectRedirectReason={type=integer}, dateTimeImport={type=date}, origMediaCap_maxFramesPerPacket={type=integer}, origMediaCap_g723BitRate={type=integer}, dateTimeConnect={type=integer}, joinOnBehalfOf={type=integer}, authorizationLevel={type=integer}, origVideoTransportAddress_Port={type=integer}, destConversationId={type=integer}, destMediaTransportAddress_Port={type=integer}, origVideoCap_Codec_Channel2={type=integer}, origVideoTransportAddress_Port_Channel2={type=integer}, destVideoChannel_Role_Channel2={type=integer}, lastRedirectRedirectOnBehalfOf={type=integer}, destSpan={type=integer}, destDTMFMethod={type=integer}, lastRedirectingRoutingReason={type=integer}, origMediaCap_Bandwidth={type=integer}, origCause_location={type=integer}, dateTimeDisconnect_formatted={type=date}, destVideoCap_Codec_Channel2={type=integer}, globalCallID_callId={type=integer}, origLegCallIdentifier={type=integer}, dateTimeOrigination={type=integer}, origRoutingReason={type=integer}, wasCallQueued={type=integer}, dateTimeConnect_formatted={type=date}, destVideoTransportAddress_Port={type=integer}, origVideoChannel_Role_Channel2={type=integer}, origCalledPartyRedirectOnBehalfOf={type=integer}, destVideoCap_Bandwidth_Channel2={type=integer}, destPrecedenceLevel={type=integer}, destVideoCap_Resolution={type=integer}, destMediaCap_Bandwidth={type=integer}, destVideoTransportAddress_IP={type=integer}, origVideoCap_Bandwidth_Channel2={type=integer}, cdrRecordType={type=integer}, calledPartyPatternUsage={type=integer}, origPrecedenceLevel={type=integer}, origVideoCap_Resolution_Channel2={type=integer}, origCause_value={type=integer}, globalCallID_callManagerId={type=integer}, dateTimeDisconnect={type=integer}, origMediaTransportAddress_IP={type=integer}, origMediaTransportAddress_Port={type=integer}, destMediaCap_payloadCapability={type=integer}, destVideoCap_Codec={type=integer}, duration={type=integer}, origVideoCap_Bandwidth={type=integer}, destCause_value={type=integer}, destMediaTransportAddress_IP={type=integer}, origDeviceName={index=not_analyzed, type=string}, destDeviceName={index=not_analyzed,type=string}, OutgoingProtocolID={type=integer}, origVideoTransportAddress_IP_Channel2={type=integer}, origNodeId={type=integer}, origCallTerminationOnBehalfOf={type=integer}, origIpAddr={type=integer}, destNodeId={type=integer}, origCalledPartyRedirectReason={type=integer}, origVideoCap_Resolution={type=integer}, destMediaCap_maxFramesPerPacket={type=integer}, destCallTerminationOnBehalfOf={type=integer}, destVideoCap_Bandwidth={type=integer}, origMediaCap_payloadCapability={type=integer}, callSecuredStatus={type=integer}, origVideoTransportAddress_IP={type=integer}, origConversationId={type=integer}, destVideoCap_Resolution_Channel2={type=integer}, destIpAddr={type=integer}, currentRoutingReason={type=integer}, totalWaitTimeInQueue={type=integer}}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters: [cucm-cdr : {_source={enabled=true}, properties={destVideoTransportAddress_Port_Channel2={type=integer}, destVideoTransportAddress_IP_Channel2={type=integer}, origVideoCap_Codec={type=integer}, IncomingProtocolID={type=integer}, origSpan={type=integer}, destMediaCap_g723BitRate={type=integer}, destLegIdentifier={type=integer}, lastRedirectRedirectReason={type=integer}, dateTimeImport={type=date}, origMediaCap_maxFramesPerPacket={type=integer}, origMediaCap_g723BitRate={type=integer}, dateTimeConnect={type=integer}, joinOnBehalfOf={type=integer}, authorizationLevel={type=integer}, origVideoTransportAddress_Port={type=integer}, destConversationId={type=integer}, destMediaTransportAddress_Port={type=integer}, origVideoCap_Codec_Channel2={type=integer}, origVideoTransportAddress_Port_Channel2={type=integer}, destVideoChannel_Role_Channel2={type=integer}, lastRedirectRedirectOnBehalfOf={type=integer}, destSpan={type=integer}, destDTMFMethod={type=integer}, lastRedirectingRoutingReason={type=integer}, origMediaCap_Bandwidth={type=integer}, origCause_location={type=integer}, dateTimeDisconnect_formatted={type=date}, destVideoCap_Codec_Channel2={type=integer}, globalCallID_callId={type=integer}, origLegCallIdentifier={type=integer}, dateTimeOrigination={type=integer}, origRoutingReason={type=integer}, wasCallQueued={type=integer}, dateTimeConnect_formatted={type=date}, destVideoTransportAddress_Port={type=integer}, origVideoChannel_Role_Channel2={type=integer}, origCalledPartyRedirectOnBehalfOf={type=integer}, destVideoCap_Bandwidth_Channel2={type=integer}, destPrecedenceLevel={type=integer}, destVideoCap_Resolution={type=integer}, destMediaCap_Bandwidth={type=integer}, destVideoTransportAddress_IP={type=integer}, origVideoCap_Bandwidth_Channel2={type=integer}, cdrRecordType={type=integer}, calledPartyPatternUsage={type=integer}, origPrecedenceLevel={type=integer}, origVideoCap_Resolution_Channel2={type=integer}, origCause_value={type=integer}, globalCallID_callManagerId={type=integer}, dateTimeDisconnect={type=integer}, origMediaTransportAddress_IP={type=integer}, origMediaTransportAddress_Port={type=integer}, destMediaCap_payloadCapability={type=integer}, destVideoCap_Codec={type=integer}, duration={type=integer}, origVideoCap_Bandwidth={type=integer}, destCause_value={type=integer}, destMediaTransportAddress_IP={type=integer}, origDeviceName={index=not_analyzed, type=string}, destDeviceName={index=not_analyzed, type=string}, OutgoingProtocolID={type=integer}, origVideoTransportAddress_IP_Channel2={type=integer}, origNodeId={type=integer}, origCallTerminationOnBehalfOf={type=integer}, origIpAddr={type=integer}, destNodeId={type=integer}, origCalledPartyRedirectReason={type=integer}, origVideoCap_Resolution={type=integer}, destMediaCap_maxFramesPerPacket={type=integer}, destCallTerminationOnBehalfOf={type=integer}, destVideoCap_Bandwidth={type=integer}, origMediaCap_payloadCapability={type=integer}, callSecuredStatus={type=integer}, origVideoTransportAddress_IP={type=integer}, origConversationId={type=integer}, destVideoCap_Resolution_Channel2={type=integer}, destIpAddr={type=integer}, currentRoutingReason={type=integer}, totalWaitTimeInQueue={type=integer}}}]"}},"status":400}[root@22bea1c5078d /]#

Can you please help or make any suggestions?

Thank you,

Daryl

How do you handle cdr cleanup?

The cdr stuff worked for a few days. I was doing a find command with -delete after 10 hours to keep the cdr's clean.

How do you handle the cleanup?

root@ubuntulog:/home/cdr/cdr# ls | grep -c Stand
7044
root@ubuntulog:/home/cdr/cdr# lsof | grep -c cdr
394960
root@ubuntulog:/home/cdr/cdr# lsof | grep -c Stand
331632
root@ubuntulog:/home/cdr/cdr#

Code update for 6 version

Hi!
I'm new to ELK, trying to setup my new customization for cucm statistics. Have some problems on 6.2.4 due to changes of architecture, such as removal of mapping types.
I did some changes to get indexes working, but not very well. It would be great if author refurbish the code from his view and make more instructions how to adapt this config to the new installation, because this could be very valuable for many people.

Here is my config changes:

cucm-cdr.conf

input {
        file {
                path =>"/path-to-cdr/cdr*"
                start_position => "beginning"
        add_field => { "cucm_clustername" => "CUCM_Cluster_1" }
        }
}

filter {
                mutate { replace => { "type" => "cucm-cdr" } }
...
}


output {
	elasticsearch {
		index => "cucm-cdr-%{+YYYY.MM.dd}"
        	hosts => localhost 
		}	
}

cucm-cmr.conf

input {
        file {
                path =>"/path-to-cdr/cmr*"
                start_position => "beginning"
        add_field => { "cucm_clustername" => "CUCM_Cluster_1" }
        }
}

filter {
                mutate { replace => { "type" => "cucm-cmr" } }
...
}

output {
	elasticsearch {
		index => "cucm-cmr-%{+YYYY.MM.dd}"
        	hosts => localhost 
		}	

}

cucm-cdr.json

...
  "mappings": {
        "doc": {
            "_source":{"enabled":true},
            "properties": {
				"type" : { "type" : "keyword" },
...
				"origDeviceName" : { "type" : "keyword" , "index": "true"},
				"destDeviceName" : { "type" : "keyword" , "index": "true"},
...

cucm-cmr.json

...
    "mappings": {
        "doc": {
            "_source":{"enabled":true},
            "properties": {
				"type" : { "type" : "keyword" },
...
				"deviceName" : { "type" : "keyword" , "index": "true"},
...