joshuar/pingbeat

DEPRECATED. Pingbeat sends ICMP packets and stores the RTT in Elasticsearch or other outputs supported by libbeat.

pingbeat's Issues

Missing 'loss' field in pingbeat output

in 1.0-beta, there is a 'loss:boolean' field that can capture packet loss. It seems this field is no longer present with 5.4.

This field is quite useful when detecting network instability. Could this field be added back?

ppingbeat panics with stack trace

# ./pingbeat -c pingbeat.yml
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xaf5ae1]

goroutine 120 [running]:
github.com/joshuar/pingbeat/beater.RecvPings(0xb897, 0xc4201fa3f0, 0xc420183170, 0xc420188640)
	/home/tigran/eProjects/go/src/github.com/joshuar/pingbeat/beater/pingbeat.go:207 +0x1e1
created by github.com/joshuar/pingbeat/beater.(*Pingbeat).Run
	/home/tigran/eProjects/go/src/github.com/joshuar/pingbeat/beater/pingbeat.go:116 +0xa0c

my config:

pingbeat:
  # Defines how often a ping is sent to a target
  period: 1s
  # Whether to send pings over IPv4
  useipv4: true
  # Whether to send pings over IPv6
  useipv6: true
  targets:
    - name: "cloud01"
      tags: "cloud01"
      desc: "cloud host 01"
    - name: "cloud02"
      tags: "cloud02"
      desc: "cloud host 02"
    - name: "cloud03"
      tags: "cloud03"
      desc: " cloud host 03"
    - name: "cloud04"
      tags: "cloud04"
      desc: "cloud host 04"
    - name: "cloud05"
      tags: "ycloud05"
      desc: "cloud host 05"
    - name: "cloud06"
      tags: "cloud06"
      desc: "cloud host 06"

output.elasticsearch:
  template.enabled: false
  hosts: ["es-lab:9200"]

Problem with make

vendor/github.com/elastic/beats/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go:109: req.WithContext undefined (type *http.Request has no field or method WithContext)
vendor/github.com/elastic/beats/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go:175: unknown http.Transport field 'DialContext' in struct literal
vendor/github.com/elastic/beats/libbeat/scripts/Makefile:74: recipe for target 'pingbeat' failed
how to fix these errors ??

Stuck on fetching updates for github.com/oschwald/geoip2-golang

This repo looks awesome and I really want to use it for an ELK stack project at UCLA I'm working on. However, in following the current installation instructions on a CentOS 6 node with Go installed, the installation is stuck at "[INFO] --> Fetching updates for github.com/oschwald/geoip2-golang." after running "make install."

e.g.:

[INFO] Downloading dependencies. Please wait...

[INFO] --> Fetching updates for github.com/elastic/beats.

[INFO] --> Fetching updates for golang.org/x/net.

[INFO] --> Fetching updates for gopkg.in/go-playground/pool.v3.

[INFO] --> Fetching updates for github.com/oschwald/geoip2-golang.

Then, when running ./glide install manually (in the Makefile), I get the following error:

[ERROR] Update failed for gopkg.in/go-playground/pool.v3: Unable to retrieve checked out version

[ERROR] Failed to do initial checkout of config: Unable to retrieve checked out version

README.md is outdated rendering installation extremely arduous

There are several issues with the installation steps in README.md:

• README.md delegates requirements to an external source, which has been changed -- there is no longer a dependencies section in libbeat's CONTRIBUTING.md file.
• running make install in the repository does not yield the correct results: make: *** No rule to make target `install'. Stop.

There are more problems with the Makefile:

• It references ../../elastic/beats/libbeat/scripts/Makefile(this is the correct path, when repositories are fully fetched with go get), but by default it assumes ./vendor/github.com/elastic/beats/libbeat/scripts/Makefile (funny enough, there is a target requiring glide in the Makefile, but this requirement is not documented anywhere)
• Docker seems to be a dependency also, but this is not documented anywhere.

I've never had such problems building a binary from source...

Missing target.description

I'm trying to use the default configuration and I'm expecting target.description in ES but it's not available.

...
pingbeat:
  # Defines how often a ping is sent to a target
  period: 1s
  # Whether to send pings over IPv4
  useipv4: true
  # Whether to send pings over IPv6
  useipv6: false
  targets:
    - name: "127.0.0.1"
      tags: "localhost"
      desc: "there's no place like home"
...

Running pingbeat -e -d "*" -c /etc/pingbeat/pingbeat.yml. Publish doesn't show the field either.

...
2018/10/04 14:32:42.891583 client.go:214: DBG  Publish: {
  "@timestamp": "2018-10-04T14:32:42.891Z",
  "beat": {
    "hostname": "my-host",
    "name": "my-host",
    "version": "5.6.10"
  },
  "logzio_codec": "json",
  "rtt": 0.036898,
  "target": {
    "addr": "127.0.0.1",
    "name": "127.0.0.1",
    "tags": [
      "localhost"
    ]
  },
  "token": "aslkdakjaslasdklsaldjaslkjdlasjdalsjdlasjdlaksjd",
  "type": "my-host-pingbeat"
}
...

What am I missing?

Can't update

Hi @joshuar !

I've noticed you've pushed a few changed to the repo, so I figured I'd try and update.

However, it's not working currently. I'm getting the following :

~$ go get -u github.com/joshuar/pingbeat
# cd /opt/go/src/github.com/joshuar/pingbeat; git submodule update --init --recursive
No submodule mapping found in .gitmodules for path 'vendor/github.com/elastic/libbeat'
package github.com/joshuar/pingbeat: exit status 1

Thanks !

Getting error

So far so good, only getting this error.

Any ideas on what this could be?

2015/11/09 09:38:52.572100 preprocess.go:42: ERR Publishing event failed: Missin
g '@timestamp' field from event

Rewrite Target struct

Currently the target struct was written with some grand plans in mind that I didn't get around to. It might be more efficient to rewrite the target struct so it has a map with the IP address as a key for fast lookup of details based on a given IP address.

Getting error: No outputs are defined. Please define one under the output section.

On running pingbeat getting the following error

$ pingbeat -e -c ~/pingbeat.yml -d "publish"
2018/03/16 09:14:30.297381 beat.go:297: INFO Home path: [/home/saleem.a/source/link_statistics] Config path: [/home/saleem.a/source/link_statistics] Data path: [/home/saleem.a/source/link_statistics/data] Logs path: [/home/saleem.a/source/link_statistics/logs]
2018/03/16 09:14:30.297422 beat.go:192: INFO Setup Beat: pingbeat; Version: 5.6.9
2018/03/16 09:14:30.297436 publish.go:280: INFO No outputs are defined. Please define one under the output section.
2018/03/16 09:14:30.297451 beat.go:346: CRIT Exiting: error initializing publisher: No outputs are defined. Please define one under the output section.
Exiting: error initializing publisher: No outputs are defined. Please define one under the output section.

Any ideas why i am getting this error?

Cannot find package 'libbeat'

Hi @joshuar

When I try to make install, I get the following error trace. Am I meant to have certain things set up before running pingbeat or will pingbeat take care of all dependencies? Am I missing something.

(Sorry I am a total noob to Golang)

PID file

Hi,

I use Monit to monitor daemons running on my systems. It needs a PID file to be able to check whether a process is running or not.

Could you please provide an option to generate a PID file when pingbeat starts ?

Thanks.

target_name is analyzed in provided index template

The provided index template for PingBeat has the target_name analyzed which splits hostnames on dashes (e.g. abc-xyz.domain.tld will be split in abc and xyz.domain.tld). This prevents proper aggregation on hostnames. See:

elastic/elasticsearch#16706

I cannot think of a reason to analyze the target_name. Did you have anything in mind?

If not, may I suggest it be changed for others not to run into a similar issue.

Thanks for the great work. I'm actually using your beat as a model for another one I am creating. Your code is very appreciated.

Cheers,

Marco

target_name.raw is not getting data anymore

Just build the master release and now i don't get the dns in target_name.raw anymore, only in target_name which is analyzed and can't be used. how can i fix this ?

Also the predefined dashboards aren't available anymore.

Thanks.

I want to get timeout in pingbeat output

I wanna get timeout Ping in pingbeat output

but, Pingbeat is Removing timeout ping data.

How can I get that?

Problem with make install

make: *** [copy-vendor] error 127

GeoIP Lookup of targets

Would be nice to enrich each ping result with geoip data of the target.

Upgrade libbeat to beats

The libbeat library and sub module you're using have been moved to https://github.com/elastic/beats. I'm opening this issue to track the migration of this code to using the new library location. If I don't hear back in a few days I'll begin work on this.

Rejecting mapping update

ES ver.7.5.0

2020/06/06 14:19:19.744526 client.go:447: WARN Can not index event (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [pingbeat-2020.06.06] as the final mapping would have more than 1 type: [_doc, doc]"}

Beats Heartbeat collaboration

It would seem there is an official experimental beat being worked on here https://github.com/elastic/beats/tree/master/heartbeat that looks very similiar to what pingbeat is doing. Have you looked at this beat and if so what are your thought on it? I love using pingbeat, but if there is an official beat that does something similiar I will likely choice that option.

Don't require glide for make; vendor dependencies

Hey, sorry for all of the attention on this lately. I started using this and wanted to give back a bit where possible.

I wanted to open this issue to start a discussion about the use of vendoring in this repo. So the idea behind vendoring is that for almost all cases in go repos, a users should be able to git pull or go get a repo and build it for their machine using go build or go install exclusively. This helps reduce tooling fatigue and "keep it simple". The issue is that if a library isn't living constantly on the bleeding edge of its dependencies, you could get build inconsistencies using go exclusively. That's where tools like glide come in.

Glide is really great for dependency management, and using it with check-in vendoring means only people upgrading dependencies need to have it installed.

My proposal is 3(4) fold:

• Modifications to the Makefile to not depend on glide and possibly focus the make targets
• Utilize vendoring and check in dependencies to keep all dependent source in this repo
• Change documentation to encourage use of go build exclusively for building a binary for the system being built on
• (4th) This may open opportunities in the CI build to reduce complexity.

Let me know what you think. If you want to see what some of these changes might look like, I can get some PR together to show what I mean.

Why is the username/password to elasticsearch deprecated?

Hi,

Reading through the documentation set out here and it says that the username/password fields to elasticsearch are deprecated. Is there a correct way to specify the username/password to elasticsearch?

Kibana 4 Sample Dashboard

Create a sample dashboard for Kibana 4 for pingbeat

Installation instructions for Logstash 2.1

Would you be able to update the installation info to outline how a non-developer could get this running (possible ./pluign method).
The go command doesn't seem to do anything descernable on my ubuntu box and not sure where it needs to be run and what needs to be done running the command.

I have a customer demo this would be excellent for along side packetbeat, topbeat and filebeat.

Too many IPs result in artificial loss docs?

I was using a config that included www.facebook.com and www.google.com, along with a us-east-1 elb dualstack hostname, and all of this expands to quite an IP list once you figure in IPv6. I left that running for a day (with 12 sec interval), and noticed a ton of loss docs. Right after I would notice a loss doc, I would try manually pinging the address and it would be very responsive. So my graphs weren't giving me a real picture of connectivity.

Reducing the configuration to a simple set of about 5-6 IPs has eliminated the loss docs, including from ones that were reporting consistently before. The top graph is a median agg over the RTT values, the bottom is a count agg over loss: true. With the simpler config, they go to absent.

New config:

input:
  period: 12
  privileged: true
  useipv4: true
  useipv6: true
  targets:
    comcast:
      - xe-5-2-0-sur01.blairblvd.tn.nash.comcast.net
    comcast dns:
      - cdns01.comcast.net
    google dns:
      - google-public-dns-a.google.com
    level3:
      - 4.2.2.2

Cannot publish events to es2x because of '.' character in field name

Hi Joshua,

I'm not able to index event in elasticsearch v 2.4.1. The ping beat is successfully connecting to es.

Detected Elasticsearch 2.x. Automatically selecting the 2.x version of the template 2017/08/16 12:54:02.146438 beat.go:221: INFO pingbeat start running. 2017/08/16 12:54:02.146470 pingbeat.go:79: INFO pingbeat is running! Hit CTRL-C to stop it. 2017/08/16 12:54:02.147816 pingbeat.go:107: INFO Using ip4:icmp connection 2017/08/16 12:54:02.992963 client.go:588: INFO Elasticsearch template with name 'pingbeat' loaded
But after that it throw lots of warnings like:
client.go:442: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"Field name [target.addr] cannot contain '.'"}

I think ProcessPing function from pingbeat.go should be adjusted in order to adapt the event fields based on the es version (e.g replace '.' with '_' in the field name for es2x ).
event := common.MapStr{ "@timestamp": common.Time(time.Now().UTC()), "type": "pingbeat", "target.name": name, "target.addr": ping.Target, "target.tags": tags, "loss": true, "reason": ping.LossReason, }

I've tried to recompile the pingbeat but it seems that go libraries for beat have been refactored and I've got a lot of errors :D.

Regards,
Adrian.

Packet loss

I beleve it will simply not log if the packet is not answered within the max rtt. Would it be posible to register it as lost or still writing it but with rtt of 999 or something like that.

On Windows, "Ping failed socket" errors

Just want to report an issue.

Pingbeat on Windows does not work. When starting the service the logs fill up immediately with the following error:

"CRIT Error: Ping failed socket: The requested protocol has not been configured into the system, or no implementation for it exists."

Error occurs on Windows 10 build 1511 PC, Windows Server 2008 R2 with SP1 server (fully updated), and on Windows 2012 R2 server (fully updated).

Pingbeat was built today on my Windows 10 PC. Followed standard go installation and git install instructions. I'm using the YML config from the sample provided just trying to ping google.com as a test. I tried adding in my own internal IP addresses for a few servers in the network but the same error occurs.

Index template no longer needed?

Just got this up and running. Very cool :)

One diversion was this comment from the README:

If intending on using the Elasticsearch output, you should add a new index template using the supplied one, for example with:

curl -XPUT  /_template/pingbeat -d @/path/to/pingbeat.template.json

Running pingbeat with the debugger then showed this error:

2016/07/02 05:01:19.284452 client.go:173: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"wrong value for index [true] for field [hostname]"}

After deleting the template and the index and running pingbeat again, everything worked fine. It appears that libbeat has changed to use the field mappings from the fields.yml file, correct? If so, the README should probably be updated to avoid confusion.

run error: "Exiting: privileged specified but not running with privileges" in windows

os: win7
pingbeat: 5.4.1
error: Exiting: privileged specified but not running with privileges