Issue: Node count doesn't increase when new nodes come online.

[skx]

Issue: Node count doesn't increase when new nodes come online.

[skx]

Just to recap this is the process-flow:

• Puppet runs on a node.
• The results of the run are sent to the puppet-master.
• The puppet-master stores the report locally.
  • And sends a copy of that report to the puppet-summary server.
• The puppet-summary server writes the report to its local SQLite database.
  • From which all the web-pages are generated.

Your problem? You say node-count doesn't change. Does the host appear at all?

If data doesn't make it to the puppet-summary database then either:

• The puppetmaster didn't send it.
• Or the puppet-master sent it, but my server rejected it (failed to parse, out of memory, permission denied writing to the sqlite database, etc).

I'd suggest looking at the logs to try to pin down which part fails.

Does that help at all? Do existing nodes continue to update when they run? (i.e. is everything broken. Or is it just new nodes which are failing.)

[tnail]

Thanks for the troubleshooting direction. I do see updates getting into puppet summary (at least I see completed calls to /upload from the logs, see below), but these aren’t making it into the database for some reason. Puppet master configuration (in /etc/puppetlabs/puppet/puppet.conf): [master] . . . ## Puppet-summary report state reports = store, http reporturl = http://localhost:3001/upload Example log entries (not as useful, as I’m not seeing any host information , just connects from localhost…): Mar 18 10:36:33 cm1 puppet-summary: 127.0.0.1 - - [18/Mar/2019:10:36:33 -0500] "POST /upload HTTP/1.1" 200 0 Mar 18 10:36:33 cm1 puppet-summary: 127.0.0.1 - - [18/Mar/2019:10:36:33 -0500] "POST /upload HTTP/1.1" 200 0 Mar 18 10:36:35 cm1 puppet-summary: 127.0.0.1 - - [18/Mar/2019:10:36:35 -0500] "POST /upload HTTP/1.1" 200 0 But I query the database and get nada. puppetserver]# sqlite3 /opt/puppetlabs/server/data/puppetserver/ps.db SQLite version 3.7.17 2013-05-20 00:56:22 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite> .table reports sqlite> select fqdn from reports; sqlite> select * from reports ...> ; sqlite> .quit So it’s not getting into the DB, once it’s received by puppet-summary. Question: Could starting the binary via systemd (on RHEL7) at boot time interfere with the communication to the database? If I start this from the command-line it seems to work. I saved off the DB to a backup and restarted it with: puppetserver]# su puppet -s /bin/bash -c /path/to/go/bin/puppet-summary serve and I see updates to the server list as hosts check in to the master. So I guess I have my answer. Next question is “Why?”. Any guidance for starting this as a service in systemd? Thanks, -=Tom Nail

[skx]

OK I think that your problem is that when you start it manually you end up with ps.db in the current working directory, and when you start via systemd you're writing to a different location.

If that is the case you can solve the problem either by adding the -db-path argument to serve (as per puppet-summery help serve, or by explicitly setting the current directory.

My systemd unit looks like this, and works fine - note that I set a user which you probably don't have (you seem to be using puppet) as well as the current directory:

  root@master ~ # cat /lib/systemd/system/puppet-summary.service 
  [Unit]
  Description=puppet-summary service

  [Service]
  WorkingDirectory=/srv/puppet-summary
  User=p-s
  ExecStart=/srv/puppet-summary/puppet-summary serve
  Restart=always
  StartLimitInterval=2
  StartLimitBurst=20
  PrivateTmp=yes
  RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

  [Install]
  WantedBy=multi-user.target

Does that help things? If not I'd probably run find / -name ps.db -print to see if I'm write about different locations.

[tnail]

Okay, yes, my systemd unit is using puppet as it’s user, and I cottoned onto the –db-file argument at startup, but I think the WorkingDirectory parameter is what may be missing. Here is my current config: [Unit] Description=Puppet-summary dashboard service Requires=puppetserver.service httpd.service After=network.target [Service] Type=simple User=puppet ExecStart=/opt/puppetlabs/server/data/puppetserver/go/bin/puppet-summary serve -db-file /opt/puppetlabs/server/data/puppetserver/ps.db ExecStop=/bin/kill $MAINPID [Install] WantedBy=multi-user.target Any concerns about using the “ExecStop” command listed above? Thanks, -=Tom Nail

[skx]

Okay, yes, my systemd unit is using puppet as it’s user, and I cottoned onto the –db-file argument at startup, but I think the WorkingDirectory parameter is what may be missing.

If you're using a full-path then the working directory shouldn't matter. It's just that if you don't use an argument it assumes ./ps.db so the cwd/pwd matters.

Any concerns about using the “ExecStop” command listed above?

Not that I can think of, no. I found stop/restart works fine without anything explicit being written for the stop-command, but having it shouldn't matter.

[tnail]

Great, thanks again for the help troubleshooting! I really like the application. It’s orders of magnitude faster than our puppetdb installation and perfect for our use case. Great job! Cheers, -=Tom Nail

[tnail]

Also, sorry about the excess verbiage. Hope you don't mind that I edited my responses to get rid of the email auto-quoting and generally clean up the thread.

[skx]

No worries, and I'm glad you like it. It scales pretty well, but it does do less than the puppetdb so I guess that the better performance is somewhat to be expected.

Anyway I'll close this now if you're happy that things are working correctly, and as expected.