Puppet package "ensure => latest" doesn't always work

Question

I have a unique situation where the Puppet package ensure => latest directive only worked on some of my systems. Out of 30 servers, the packages updated properly on all but 4. I'm trying to get major versions of the software suite from < 9.40 to 9.50. I could specify a hard level, but I'd run into the multiple architecture wildcard issue.

Manifest code snippet:

# Install packages
$spppackages = [ "hp-snmp-agents", "hpssa", "hp-health", "hp-smh-templates", "hpsmh", "hpssacli", "hponcfg", ]

package { $spppackages:
  ensure  => latest,
  require => Yumrepo['HP-spp'],
}

Mcollective status:

mco package hp-snmp-agents status

Summary of Ensure:

   9.50-2564.40.rhel6 = 14
   9.50-2564.34.rhel5 = 9
   9.40-2506.37.rhel6 = 4 <== This is what I'm trying to fix!
   9.50-2564.35.rhel5 = 1

I've tried debugging this on the affected systems with:

puppet apply --verbose --debug --execute 'package { hp-snmp-agents: ensure => latest }'

Resulting in:

Info: Applying configuration version '1393411670'
Debug: Prefetching yum resources for package
Debug: Executing '/bin/rpm --version'
Debug: Executing '/bin/rpm -qa --nosignature --nodigest --qf '%{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE} %{ARCH} :DESC: %{SUMMARY}\n''
Error: Could not prefetch package provider 'yum': invalid byte sequence in US-ASCII
Debug: Executing '/bin/rpm -q hp-snmp-agents --nosignature --nodigest --qf '%{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE} %{ARCH} :DESC: %{SUMMARY}\n''
Debug: Finishing transaction 11762680
Debug: Storing state
Debug: Stored state in 0.09 seconds

Running yum list updates hp-snmp-agents shows that the package can be updated by yum and there's a newer version available:

Installed Packages
hp-snmp-agents.x86_64                                   9.40-2506.37.rhel6                                   @HP-spp
Available Packages
hp-snmp-agents.x86_64                                   9.50-2564.40.rhel6                                   HP-spp 

It seems like the problem could potentially be the rpm version query format. Running that by hand yields:

# /bin/rpm -q hp-snmp-agents --nosignature --nodigest --qf '%{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE} %{ARCH} :DESC: %{SUMMARY}\n'
hp-snmp-agents 0 9.40 2506.37.rhel6 x86_64 :DESC: Insight Management Agents(SNMP) for HP ProLiant Systems

If I ensure => 9.50-xxx on the hp-snmp-agents package, I can force this to work on the stubborn systems. Even inputting a bogus version number there seems to jumpstart the catalog build, and subsequent runs of the puppet agent work and update properly. I'm just curious why this worked seamlessly on 26 of 30 servers and needed massaging on the rest.

Note, I did yum clean all via Mcollective as an initial troubleshooting step

Answer 1

I believe the relevant problem is this:

Debug: Executing '/bin/rpm -qa --nosignature --nodigest --qf '%{NAME} %|EPOCH?{%{EPOCH}}:{0}| %{VERSION} %{RELEASE} %{ARCH} :DESC: %{SUMMARY}\n''
Error: Could not prefetch package provider 'yum': invalid byte sequence in US-ASCII

Which is talked about in this bug: https://tickets.puppetlabs.com/browse/PUP-736

Basically, somewhere in that rpm -qa ... output, you're getting a UTF-8 (or other non-ASCII) character, and that's causing puppet to think that the "yum" provider is no good. Since the "rpm" fallback provider has no ensure => latest support, it ignores that and you get the old version.

You can either upgrade to puppet 3.4.3, which includes a fix, or compare /bin/rpm -qa ... | sort output between boxes that worked right and didn't work right to find the culprit package. Changing from LANG=C to LANG=en_US.UTF-8 (or any other valid UTF-8 LANG) should temporarily resolve the issue, too.