Just found another VM where this happened: liwa3-2.linkwatcher.eqiad1.wikimedia.cloud

Change #1091733 merged by Majavah:

[operations/puppet@production] keepalived::failover: Support IPv6

https://gerrit.wikimedia.org/r/1091733

Change #1091732 merged by Majavah:

[operations/puppet@production] keepalived: Split failover config template to new class

https://gerrit.wikimedia.org/r/1091732

With BBU:

root@backup1012:~$ ./storcli64 show all J
{
"Controllers":[
{
        "Command Status" : {
                "CLI Version" : "007.3103.0000.0000 Aug 22, 2024",
                "Operating system" : "Linux 6.1.0-26-amd64",
                "Status Code" : 0,
                "Status" : "Success",
                "Description" : "None"
        },
        "Response Data" : {
                "Number of Controllers" : 1,
                "Host Name" : "backup1012",
                "Operating System " : "Linux 6.1.0-26-amd64",
                "System Overview" : [
                        {
                                "Ctl" : 0,
                                "Model" : "SAS3908",
                                "Ports" : 8,
                                "PDs" : 24,
                                "DGs" : 1,
                                "DNOpt" : 0,
                                "VDs" : 1,
                                "VNOpt" : 0,
                                "BBU" : "Opt",
                                "sPR" : "On",
                                "DS" : "1&2",
                                "EHS" : "Y",
                                "ASOs" : 4,
                                "Hlth" : "Opt"
                        }
                ],
                "ASO" : [
                        {
                                "Ctl" : 0,
                                "Cl" : "X",
                                "SAS" : "U",
                                "MD" : "U",
                                "R6" : "U",
                                "WC" : "U",
                                "R5" : "U",
                                "SS" : "U",
                                "FP" : "U",
                                "Re" : "X",
                                "CR" : "X",
                                "RF" : "X",
                                "CO" : "X",
                                "CW" : "X",
                                "HA" : "X",
                                "SSHA" : "X"
                        }
                ]
        }
}
]
}

Currently not supported by the pyyaml https://github.com/yaml/pyyaml/issues/90

Change #1091733 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] keepalived::failover: Support IPv6

https://gerrit.wikimedia.org/r/1091733

Change #1091732 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] keepalived: Split failover config template to new class

https://gerrit.wikimedia.org/r/1091732

Change #1091731 merged by Jcrespo:

[operations/puppet@production] backup: Move Dell bacula hosts to mediabackups

https://gerrit.wikimedia.org/r/1091731

Change #1091731 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] backup: Move Dell bacula hosts to mediabackups

https://gerrit.wikimedia.org/r/1091731

Change #1091249 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] resolvconf: don't update resolv.conf with 0 nameservers

https://gerrit.wikimedia.org/r/1091249

The issue is resolved, I created this task to track it in case it happens again.

As a workaround, is it possible to use a greasemonkey script (browser addon) on Android to make the "m." be removed from the URL when the user agent is set to desktop?

Summary of an IRC chat between me, Jayme and Matthew:

Perhaps relevantly, I was screenshotting the BMC storage page on another one of these hosts:

In T377853#10251006, @jcrespo wrote:

perccli and storecli are not exactly the same either, existing script fails with:

Failed to execute ['/usr/local/lib/nagios/plugins/get-raid-status-perccli']: KeyError 'BBU_Info'

Nope, megacli doesn't work. That's the one option I tried first, before going into this rabbit hole. 0:-)

does megacli work? That's the tool (from the megacli package) that I use to interact with the RAID controller on the existing config-J systems (Dell).

perccli and storecli are not exactly the same either, existing script fails with:

Failed to execute ['/usr/local/lib/nagios/plugins/get-raid-status-perccli']: KeyError 'BBU_Info'

After testing on older hosts, storecli seems to work on older hosts from a different vendor, so either there is something broken with this host in particular (I would need a second new host with a RAID controller to check) or we should upgrade the cli to use this other tool, that seems to work for both vendors.

Haven't seen widespread problems with export_smart_data_dump.service in the last 90d

Change #719100 abandoned by Muehlenhoff:

[operations/puppet@production] puppetmaster: puppet prometheus reporting

Reason:

No longer needed

https://gerrit.wikimedia.org/r/719100

Change #1074358 merged by Muehlenhoff:

[operations/puppet@production] icinga: Enable profile::auto_restarts::service for keyholder-proxy

https://gerrit.wikimedia.org/r/1074358

Change #1074358 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] icinga: Enable profile::auto_restarts::service for keyholder-proxy

https://gerrit.wikimedia.org/r/1074358

I had a chat with Filippo, the keyholder-proxy is not the daemon that needs re-arming when restarted, so it can be done anytime without extra manual steps.

Change #724733 merged by JHathaway:

[operations/puppet@production] P:tlsproxy::instance: Drop numa_networking global

https://gerrit.wikimedia.org/r/724733