SREGroup
ActivePublic
Watch Project

Members (31)

Ladsgroup (Amir Sarabadani)
Shah of Bugs, Emir of database architecture, World-renowned rubber duck
ABran-WMF (arnaudb)
SRE
andrea.denisse (denisse)
Animal
cmooney (Cathal Mooney)
SRE (netops)
RLazarus (Reuven Lazarus) (rzl)
User
Devnull (Mark Rosenbaum)
Infrastructure Engineer
lmata (Leo Mata)
SRE
DubOSv10 (DubOSv10)
User
colewhite (cwhite)
User
AndyTan (AndyTan)
User
View All

Watchers (13)

Wunderlandmeli
User
ReaperDawn (Vulphere)
User
Muchiri124 (Edwin Muchiri Kibuti)
User
Devnull (Mark Rosenbaum)
Infrastructure Engineer
Techguru.pc (Paul Charlton)
Tech Guru
Legado_Shulgin (psiconauta)
User
Wong128hk (JWong)
from zhwikis
Davinaclare77
User
Hfbn0 (Hfbn0)
User
Zppix (Devin)
Volunteer Developer & Volunteer Support on IRC & MW
View All

Details

Description

Homepage: https://www.mediawiki.org/wiki/Wikimedia_Site_Reliability_Engineering

If you're filing a task that needs to be handled by the SRE team, add a tag from the list below. If you're not sure which one to pick, add the SRE tag and the on-duty person will redirect it as appropriate.

Cross-team tags:

Tags by team:

DC-Ops
- procurement
- ops-codfw, ops-eqiad, ops-eqsin, ops-esams, ops-ulsfo, ops-eqord, ops-eqdfw, ops-magru
Traffic
- Domains
- DNS
- PyBal
- Traffic-Icebox (do not use for new tasks)
Data-Persistence
serviceops
collaboration-services
- GitLab
- Miscweb
- Phabricator
- VRTS / Znuny
Infrastructure-Foundations
- SRE-tools
- Mail
- Packaging
- netops
- netbox
- homer
- Puppet
- Puppet CI
- Puppet-Core
- Puppet-Infrastructure
- CAS-SSO
- CFSSL-PKI
- vm-requests
- Infrastructure Security
- Keyholder
SRE Observability
- observability (do not use for new tasks)
- Observability-Alerting
- Observability-Logging
- Observability-Metrics
- Observability-Tracing
- Pontoon

Recent Activity
View All

Yesterday

VRiley-WMF closed T380564: decommission ganeti1011 / ganeti1014 as Resolved.

Sun, Nov 24, 8:13 PM · SRE, DC-Ops, ops-eqiad, decommission-hardware

VRiley-WMF claimed T380564: decommission ganeti1011 / ganeti1014.

Sun, Nov 24, 7:14 PM · SRE, DC-Ops, ops-eqiad, decommission-hardware

Sat, Nov 23

Pppery moved T355914: Change default image thumbnail size from Backlog to Blocked on development on the Wikimedia-Site-requests board.

Sat, Nov 23, 6:40 PM · Web-Team, Web Team Essential Work 2024, Design, Wikimedia-Design, Thumbor, Traffic, SRE-swift-storage, Data-Persistence, SRE, Wikimedia-Site-requests

Fri, Nov 22

ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1002 for host aux-k8s-worker2005.codfw.wmnet with OS bookworm completed:

aux-k8s-worker2005 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411222041_herron_4011077_aux-k8s-worker2005.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Fri, Nov 22, 8:59 PM · vm-requests, SRE, Kubernetes

Maintenance_bot moved T378146: Q2:rack/setup/install es204[1-6] from In progress to Done on the DBA board.

Fri, Nov 22, 8:29 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops

ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1002 for host aux-k8s-worker2005.codfw.wmnet with OS bookworm

Fri, Nov 22, 8:20 PM · vm-requests, SRE, Kubernetes

ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1002 for host aux-k8s-worker2004.codfw.wmnet with OS bookworm completed:

aux-k8s-worker2004 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221931_herron_4000086_aux-k8s-worker2004.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Fri, Nov 22, 7:47 PM · vm-requests, SRE, Kubernetes

Jhancock.wm closed T378146: Q2:rack/setup/install es204[1-6] as Resolved.

@ABran-WMF finished

Fri, Nov 22, 7:42 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops

Jhancock.wm updated the task description for T378146: Q2:rack/setup/install es204[1-6].

Fri, Nov 22, 7:37 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops

ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2045.codfw.wmnet with OS bookworm completed:

es2045 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221919_jhancock_3002599_es2045.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Updated Netbox status planned -> active
- The sre.puppet.sync-netbox-hiera cookbook was run successfully

Fri, Nov 22, 7:36 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops

ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2046.codfw.wmnet with OS bookworm completed:

es2046 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221916_jhancock_3002589_es2046.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Updated Netbox status planned -> active
- The sre.puppet.sync-netbox-hiera cookbook was run successfully

Fri, Nov 22, 7:35 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops

ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2043.codfw.wmnet with OS bookworm completed:

es2043 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221913_jhancock_3002609_es2043.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Updated Netbox status planned -> active
- The sre.puppet.sync-netbox-hiera cookbook was run successfully

Fri, Nov 22, 7:32 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops

ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2044.codfw.wmnet with OS bookworm completed:

es2044 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221909_jhancock_3002604_es2044.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Updated Netbox status planned -> active
- The sre.puppet.sync-netbox-hiera cookbook was run successfully

Fri, Nov 22, 7:27 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops

ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1002 for host aux-k8s-worker2004.codfw.wmnet with OS bookworm

Fri, Nov 22, 7:13 PM · vm-requests, SRE, Kubernetes

ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2042.codfw.wmnet with OS bookworm completed:

es2042 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221835_jhancock_2995890_es2042.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Updated Netbox status planned -> active
- The sre.puppet.sync-netbox-hiera cookbook was run successfully

Fri, Nov 22, 6:53 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops

ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2044.codfw.wmnet with OS bookworm

Fri, Nov 22, 6:50 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops

ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2043.codfw.wmnet with OS bookworm

Fri, Nov 22, 6:50 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops

ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2045.codfw.wmnet with OS bookworm

Fri, Nov 22, 6:50 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops

ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2046.codfw.wmnet with OS bookworm

Fri, Nov 22, 6:50 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops

ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1002 for host aux-k8s-worker2003.codfw.wmnet with OS bookworm completed:

aux-k8s-worker2003 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221831_herron_3991918_aux-k8s-worker2003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Fri, Nov 22, 6:45 PM · vm-requests, SRE, Kubernetes

ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2042.codfw.wmnet with OS bookworm

Fri, Nov 22, 6:17 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops

ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1002 for host aux-k8s-worker2003.codfw.wmnet with OS bookworm

Fri, Nov 22, 6:11 PM · vm-requests, SRE, Kubernetes

cmooney added a comment to T380503: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54.

Thanks @VRiley-WMF! Seems ok so far but we can make a call Monday based on if we see errors on the link or not (clean since the swap).

Fri, Nov 22, 5:59 PM · SRE, ops-eqiad, Cloud-Services, netops, DC-Ops, Infrastructure-Foundations

ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1002 for host aux-k8s-worker2002.codfw.wmnet with OS bookworm completed:

aux-k8s-worker2002 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221725_herron_3982686_aux-k8s-worker2002.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Fri, Nov 22, 5:41 PM · vm-requests, SRE, Kubernetes

VRiley-WMF added a comment to T380503: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54.

Replaced the transciever in cloudsw1-e4-eqiad et-0/0/54. Will test to see if that works. Trying to locate another one for cloudsw1-d5-eqiad et-0/0/52

Fri, Nov 22, 5:33 PM · SRE, ops-eqiad, Cloud-Services, netops, DC-Ops, Infrastructure-Foundations

ops-monitoring-bot added a comment to T380503: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54.

Icinga downtime and Alertmanager silence (ID=6b283bec-74b8-4f8c-9a46-f9f60c9c4026) set by cmooney@cumin1002 for 1:00:00 on 2 host(s) and their services with reason: replace optics on faulty WMCS link from D5 to E4

cloudsw1-d5-eqiad.mgmt,cloudsw1-e4-eqiad.mgmt

Fri, Nov 22, 5:23 PM · SRE, ops-eqiad, Cloud-Services, netops, DC-Ops, Infrastructure-Foundations

ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1002 for host aux-k8s-worker2002.codfw.wmnet with OS bookworm

Fri, Nov 22, 5:08 PM · vm-requests, SRE, Kubernetes

gerritbot added a project to T369308: Decommission clouddb2002-dev.codfw.wmnet: Patch-For-Review.

Fri, Nov 22, 5:05 PM · Patch-For-Review, SRE, ops-codfw, DC-Ops, wikitech.wikimedia.org, Data-Persistence, cloud-services-team

gerritbot added a comment to T369308: Decommission clouddb2002-dev.codfw.wmnet.

Change #1094491 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] O:openstack: Drop codfw1dev db role

https://gerrit.wikimedia.org/r/1094491

Fri, Nov 22, 5:05 PM · Patch-For-Review, SRE, ops-codfw, DC-Ops, wikitech.wikimedia.org, Data-Persistence, cloud-services-team

phaultfinder updated the task description for T380182: Inbound interface errors.

Fri, Nov 22, 5:04 PM · SRE, ops-eqiad, DC-Ops

Maintenance_bot added a project to T380473: Decommission parse20[01-20]: SRE.

Fri, Nov 22, 4:29 PM · SRE, DC-Ops, ops-codfw, decommission-hardware, serviceops

jcrespo closed T376892: Expand media backup storage available space to 960 TB per datacenter as Resolved.

This is now done. Catchup and purging is ongoing, but after that finishes, we should be able to store almost 1 PB of media backups on both datacenters. Backups will continue without problem or manual interventions in the next months.

Fri, Nov 22, 4:19 PM · Patch-For-Review, media-backups, Data-Persistence-Backup, SRE

jcrespo closed T376892: Expand media backup storage available space to 960 TB per datacenter, a subtask of T262668: WMF media storage must be adequately backed up, as Resolved.

Fri, Nov 22, 4:16 PM · media-backups, Data-Persistence-Backup, Epic, Goal, SRE, SRE-swift-storage

phaultfinder updated the task description for T379668: PDU sensor over limit.

Fri, Nov 22, 3:54 PM · SRE, DC-Ops, ops-eqiad

Stang moved T380109: Mailing list for zhwiki arbcom from Site configuration to Closed on the Chinese-Sites board.

Fri, Nov 22, 3:31 PM · SRE, Chinese-Sites, Wikimedia-Mailing-lists

Ladsgroup closed T380109: Mailing list for zhwiki arbcom as Resolved.

https://lists.wikimedia.org/postorius/lists/wikipedia-zh-arbcom.lists.wikimedia.org Creating wikis is a much bigger work, that'll take a while

Fri, Nov 22, 3:29 PM · SRE, Chinese-Sites, Wikimedia-Mailing-lists

phaultfinder updated the task description for T379668: PDU sensor over limit.

Fri, Nov 22, 3:09 PM · SRE, DC-Ops, ops-eqiad

MoritzMuehlenhoff added a comment to T376737: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged).

Ack, thanks. Either is fine with me, I can also switch them to insetup and then keep them running.

Fri, Nov 22, 2:54 PM · Patch-For-Review, SRE, Traffic, ops-magru

RobH added a comment to T376737: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged).

We were discussing this last week, and brianstormed some on https://etherpad.wikimedia.org/p/magru_server_swaps from the irc output:

Fri, Nov 22, 2:31 PM · Patch-For-Review, SRE, Traffic, ops-magru

phaultfinder updated the task description for T379668: PDU sensor over limit.

Fri, Nov 22, 2:19 PM · SRE, DC-Ops, ops-eqiad

phaultfinder updated the task description for T380182: Inbound interface errors.

Fri, Nov 22, 12:59 PM · SRE, DC-Ops, ops-eqiad

gerritbot added a comment to T379343: Create bookworm-based build host.

Change #1094387 merged by Muehlenhoff:

[operations/puppet@production] package_builder: Cleanups

https://gerrit.wikimedia.org/r/1094387

Fri, Nov 22, 12:46 PM · Patch-For-Review, Infrastructure-Foundations, SRE

gerritbot added a comment to T379343: Create bookworm-based build host.

Change #1094393 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make docker::baseimages ensurable

https://gerrit.wikimedia.org/r/1094393

Fri, Nov 22, 12:25 PM · Patch-For-Review, Infrastructure-Foundations, SRE

jcrespo added a comment to T376892: Expand media backup storage available space to 960 TB per datacenter.

Timestamp is in CET:

[12:26:36] <jinxer-wm> RESOLVED: DiskSpace: Disk space backup2011:9100:/srv/objectstorage 5.998% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=backup2011 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace

Fri, Nov 22, 11:33 AM · Patch-For-Review, media-backups, Data-Persistence-Backup, SRE

Maintenance_bot added a project to T380564: decommission ganeti1011 / ganeti1014: SRE.

Fri, Nov 22, 11:29 AM · SRE, DC-Ops, ops-eqiad, decommission-hardware

Clement_Goubert closed T376966: wikikube-worker21[56-70] implementation tracking, a subtask of T376965: Q2:rack/setup/install wikikube-worker21[56-70], as Resolved.

Fri, Nov 22, 11:26 AM · SRE, serviceops, ops-codfw, DC-Ops

jcrespo added a comment to T376892: Expand media backup storage available space to 960 TB per datacenter.

Capacity reached 94.2% and finally it is on a downward trend: 93.7% 🎉

Fri, Nov 22, 11:22 AM · Patch-For-Review, media-backups, Data-Persistence-Backup, SRE

• ayounsi updated the task description for T380050: Decommission E/F 8 Dell switches.

Fri, Nov 22, 11:19 AM · Patch-For-Review, SRE, DC-Ops, ops-eqiad

gerritbot added a project to T379343: Create bookworm-based build host: Patch-For-Review.

Fri, Nov 22, 11:18 AM · Patch-For-Review, Infrastructure-Foundations, SRE