syslog fills with "Error 1 sending the modular data", gmond keeps using socket after EINVAL

[ganglia]

syslog fills with "Error 1 sending the modular data", gmond keeps using socket after EINVAL

[creisor]

I'm seeing similar errors, but it mostly seems to be due to connection refused:

creisor@hostname ~/Documents/troubleshooting/gmond $ grep "= -1" gmond.errs  | sort | uniq -c
  14 connect(27, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
 122 lseek(25, 0, SEEK_CUR)                  = -1 ESPIPE (Illegal seek)
  61 lseek(28, 0, SEEK_CUR)                  = -1 ESPIPE (Illegal seek)
   9 write(7, "\0\0\0\200\0\0\0\16hostname\0\0\0\0\0\17net_"..., 184) = -1 ECONNREFUSED (Connection refused)
   5 write(7, "\0\0\0\200\0\0\0\16hostname\0\0\0\0\0\21proc"..., 200) = -1 ECONNREFUSED (Connection refused)
   1 write(7, "\0\0\0\200\0\0\0\16hostname\0\0\0\0\0\fredi"..., 168) = -1 ECONNREFUSED (Connection refused)
   9 write(7, "\0\0\0\200\0\0\0\16hostname\0\0\0\0\0\ncpu_"..., 168) = -1 ECONNREFUSED (Connection refused)
   2 write(7, "\0\0\0\200\0\0\0\16hostname\0\0\0\0\0\thear"..., 168) = -1 ECONNREFUSED (Connection refused)
   2 write(7, "\0\0\0\203\0\0\0\16hostname\0\0\0\0\0\17net_"..., 60) = -1 ECONNREFUSED (Connection refused)
   4 write(7, "\0\0\0\203\0\0\0\16hostname\0\0\0\0\0\21proc"..., 64) = -1 ECONNREFUSED (Connection refused)
   2 write(7, "\0\0\0\203\0\0\0\16hostname\0\0\0\0\0\ncpu_"..., 56) = -1 ECONNREFUSED (Connection refused)
   1 write(7, "\0\0\0\203\0\0\0\16hostname\0\0\0\0\0\nntp_"..., 56) = -1 ECONNREFUSED (Connection refused)
   7 write(7, "\0\0\0\204\0\0\0\16hostname\0\0\0\0\0\thear"..., 56) = -1 ECONNREFUSED (Connection refused)
   1 write(7, "\0\0\0\205\0\0\0\16hostname\0\0\0\0\0\17net_"..., 272) = -1 ECONNREFUSED (Connection refused)
   2 write(7, "\0\0\0\205\0\0\0\16hostname\0\0\0\0\0\21proc"..., 152) = -1 ECONNREFUSED (Connection refused)
   1 write(7, "\0\0\0\205\0\0\0\16hostname\0\0\0\0\0\5gexe"..., 56) = -1 ECONNREFUSED (Connection refused)
   1 write(7, "\0\0\0\205\0\0\0\16hostname\0\0\0\0\0\ncpu_"..., 172) = -1 ECONNREFUSED (Connection refused)
   1 write(7, "\0\0\0\205\0\0\0\16hostname\0\0\0\0\0\nntp_"..., 128) = -1 ECONNREFUSED (Connection refused)

FD 7 is a UDP channel to our central ganglia server

gmond   4519 ganglia    7u  IPv4          348817372      0t0        UDP 10.20.40.104:38381->10.20.43.22:8706

man 2 connect says:

ECONNREFUSED
No one listening on the remote address.

[pmigrala]

Late reply, but may help some poor soul ...

get pid of gmond:
systemctl status gmond.service

"lsof -p 20606" revealed:
gmond 20606 ganglia 7u IPv6 385361 0t0 UDP localhost:50101->localhost:8649

Notice type IPv6.
Disabling ipv6 and restarting gmond corrected my ECONNREFUSED errors.
The ganglia graphs are now being rendered.

Disabled ipv6 like this:
vi /etc/sysctl.conf

add the following:
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

restart network:
systemctl restart network

restart gmond:
systemctl restart gmond.service