1

Without changing anything to the puppet configuration suddenly, some of our hosts started with "Could not evaluate: Field 'key' is required" errors, when creating exported SSH keys but interestingly not all SSH keys fail on all hosts, here one example where keys are both successfully and unsuccessfully created:

Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for srvadx-green-7zhg.c.example.internal
Info: Applying configuration version '1511609586'
Notice: /Stage[main]/Basics::Ssh/Sshkey[srvadx-green-w5ld]/ensure: created
Info: Computing checksum on file /etc/ssh/ssh_known_hosts
Info: FileBucket got a duplicate file {md5}17787ff909a06cf0243c18cf33f1c384
Error: /Stage[main]/Basics::Ssh/Sshkey[srvadx-green-w5ld]: Could not evaluate: Field 'key' is required
Notice: /Stage[main]/Basics::Ssh/Sshkey[srvadx-green-88kp]/ensure: created
Error: /Stage[main]/Basics::Ssh/Sshkey[srvadx-green-88kp]: Could not evaluate: Field 'key' is required
Notice: /Stage[main]/Basics::Java/Exec[set-licence-selected]/returns: executed successfully
Notice: /Stage[main]/Basics::Java/Exec[set-licence-seen]/returns: executed successfully
Notice: Finished catalog run in 1.98 seconds

First I suspected a corrupt HSQLDB and deleted the whole puppetdb data directory and re-ran puppet on all hosts. First it looked promising, but soon the errors as above re-appeared.

How to debug or even better fix the problem and how to prevent it from happening again in the future?

Sebastian
  • 61
  • 3

2 Answers2

1

To help debugging/understanding what has happened, there's three things you can do (at least):

  1. Run the agent in debug mode: puppet agent -tv --debug . It's a bit of a firehose, but there might be some clues in there (it's a bit of a long shot - the problem is probably on the server
  2. Review the puppetmaster/server logs; I recommend tailing them while you run the agent on the problem node(s), so you can be sure you're seeing the output of those runs. It's entirely possible there's a clue in there (the server emits lots of warnings about things that are not-quite-right that are not visible on the agents)
  3. Go spelunking into the HSQLDB database; there should be some rows in a table somewhere related to the various ssh keys. See what info is there, particularly if some are working and some are not (compare the rows).
Craig Miskell
  • 4,086
  • 1
  • 15
  • 16
  • same issue, no clues in 1, or 2, debug mode, even with `-tv` looks same oddly enough, its only happening on one server so far, but not another. Im going spelunking, thanks! note the error seems non-volatile though, because if i delete the `authorized_keys` file, it is recreated successfully and entry is there. Hmm, just noticed, its missing the comment, so maybe thats our clue, check how your comment (resource title) is specified (mine was specified correctly, so not sure still). – Brian Thomas Dec 06 '19 at 21:36
1

In the end it worked by deleting the PuppetDB data directory and re-running puppet on all nodes. I suspect some corrupt HSQLDB files.

Sebastian
  • 61
  • 3