iBet uBet web content aggregator. Adding the entire web to your favor.
iBet uBet web content aggregator. Adding the entire web to your favor.



Link to original content: http://phabricator.wikimedia.org/tag/sre/
SRE
Page MenuHomePhabricator
Projects SRE

SREGroup
ActivePublic
Watch Project

Members (31)

Watchers (13)

Details

Description

Homepage: https://www.mediawiki.org/wiki/Wikimedia_Site_Reliability_Engineering

If you're filing a task that needs to be handled by the SRE team, add a tag from the list below. If you're not sure which one to pick, add the SRE tag and the on-duty person will redirect it as appropriate.

Cross-team tags:

Tags by team:

Recent Activity
View All

Today

VRiley-WMF closed T380564: decommission ganeti1011 / ganeti1014 as Resolved.
Sun, Nov 24, 8:13 PM · SRE, DC-Ops, ops-eqiad, decommission-hardware
VRiley-WMF claimed T380564: decommission ganeti1011 / ganeti1014.
Sun, Nov 24, 7:14 PM · SRE, DC-Ops, ops-eqiad, decommission-hardware

Yesterday

Pppery moved T355914: Change default image thumbnail size from Backlog to Blocked on development on the Wikimedia-Site-requests board.
Sat, Nov 23, 6:40 PM · Web-Team, Web Team Essential Work 2024, Design, Wikimedia-Design, Thumbor, Traffic, SRE-swift-storage, Data-Persistence, SRE, Wikimedia-Site-requests

Fri, Nov 22

ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1002 for host aux-k8s-worker2005.codfw.wmnet with OS bookworm completed:

  • aux-k8s-worker2005 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411222041_herron_4011077_aux-k8s-worker2005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Fri, Nov 22, 8:59 PM · vm-requests, SRE, Kubernetes
Maintenance_bot moved T378146: Q2:rack/setup/install es204[1-6] from In progress to Done on the DBA board.
Fri, Nov 22, 8:29 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1002 for host aux-k8s-worker2005.codfw.wmnet with OS bookworm

Fri, Nov 22, 8:20 PM · vm-requests, SRE, Kubernetes
ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1002 for host aux-k8s-worker2004.codfw.wmnet with OS bookworm completed:

  • aux-k8s-worker2004 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221931_herron_4000086_aux-k8s-worker2004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Fri, Nov 22, 7:47 PM · vm-requests, SRE, Kubernetes
Jhancock.wm closed T378146: Q2:rack/setup/install es204[1-6] as Resolved.

@ABran-WMF finished

Fri, Nov 22, 7:42 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
Jhancock.wm updated the task description for T378146: Q2:rack/setup/install es204[1-6].
Fri, Nov 22, 7:37 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2045.codfw.wmnet with OS bookworm completed:

  • es2045 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221919_jhancock_3002599_es2045.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Fri, Nov 22, 7:36 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2046.codfw.wmnet with OS bookworm completed:

  • es2046 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221916_jhancock_3002589_es2046.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Fri, Nov 22, 7:35 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2043.codfw.wmnet with OS bookworm completed:

  • es2043 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221913_jhancock_3002609_es2043.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Fri, Nov 22, 7:32 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2044.codfw.wmnet with OS bookworm completed:

  • es2044 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221909_jhancock_3002604_es2044.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Fri, Nov 22, 7:27 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1002 for host aux-k8s-worker2004.codfw.wmnet with OS bookworm

Fri, Nov 22, 7:13 PM · vm-requests, SRE, Kubernetes
ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host es2042.codfw.wmnet with OS bookworm completed:

  • es2042 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221835_jhancock_2995890_es2042.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Fri, Nov 22, 6:53 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2044.codfw.wmnet with OS bookworm

Fri, Nov 22, 6:50 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2043.codfw.wmnet with OS bookworm

Fri, Nov 22, 6:50 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2045.codfw.wmnet with OS bookworm

Fri, Nov 22, 6:50 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2046.codfw.wmnet with OS bookworm

Fri, Nov 22, 6:50 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1002 for host aux-k8s-worker2003.codfw.wmnet with OS bookworm completed:

  • aux-k8s-worker2003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221831_herron_3991918_aux-k8s-worker2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Fri, Nov 22, 6:45 PM · vm-requests, SRE, Kubernetes
ops-monitoring-bot added a comment to T378146: Q2:rack/setup/install es204[1-6].

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host es2042.codfw.wmnet with OS bookworm

Fri, Nov 22, 6:17 PM · Data-Persistence-SRE, DBA, Patch-For-Review, SRE, Data-Persistence, ops-codfw, DC-Ops
ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1002 for host aux-k8s-worker2003.codfw.wmnet with OS bookworm

Fri, Nov 22, 6:11 PM · vm-requests, SRE, Kubernetes
cmooney added a comment to T380503: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54.

Thanks @VRiley-WMF! Seems ok so far but we can make a call Monday based on if we see errors on the link or not (clean since the swap).

Fri, Nov 22, 5:59 PM · SRE, ops-eqiad, Cloud-Services, netops, DC-Ops, Infrastructure-Foundations
ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1002 for host aux-k8s-worker2002.codfw.wmnet with OS bookworm completed:

  • aux-k8s-worker2002 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411221725_herron_3982686_aux-k8s-worker2002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Fri, Nov 22, 5:41 PM · vm-requests, SRE, Kubernetes
VRiley-WMF added a comment to T380503: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54.

Replaced the transciever in cloudsw1-e4-eqiad et-0/0/54. Will test to see if that works. Trying to locate another one for cloudsw1-d5-eqiad et-0/0/52

Fri, Nov 22, 5:33 PM · SRE, ops-eqiad, Cloud-Services, netops, DC-Ops, Infrastructure-Foundations
ops-monitoring-bot added a comment to T380503: Replace optics in cloudsw1-d5-eqiad et-0/0/52 and cloudsw1-e4-eqiad et-0/0/54.

Icinga downtime and Alertmanager silence (ID=6b283bec-74b8-4f8c-9a46-f9f60c9c4026) set by cmooney@cumin1002 for 1:00:00 on 2 host(s) and their services with reason: replace optics on faulty WMCS link from D5 to E4

cloudsw1-d5-eqiad.mgmt,cloudsw1-e4-eqiad.mgmt
Fri, Nov 22, 5:23 PM · SRE, ops-eqiad, Cloud-Services, netops, DC-Ops, Infrastructure-Foundations
ops-monitoring-bot added a comment to T378987: codfw: (4x) aux-k8s-worker nodes.

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1002 for host aux-k8s-worker2002.codfw.wmnet with OS bookworm

Fri, Nov 22, 5:08 PM · vm-requests, SRE, Kubernetes
gerritbot added a project to T369308: Decommission clouddb2002-dev.codfw.wmnet: Patch-For-Review.
Fri, Nov 22, 5:05 PM · Patch-For-Review, SRE, ops-codfw, DC-Ops, wikitech.wikimedia.org, Data-Persistence, cloud-services-team
gerritbot added a comment to T369308: Decommission clouddb2002-dev.codfw.wmnet.

Change #1094491 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] O:openstack: Drop codfw1dev db role

https://gerrit.wikimedia.org/r/1094491

Fri, Nov 22, 5:05 PM · Patch-For-Review, SRE, ops-codfw, DC-Ops, wikitech.wikimedia.org, Data-Persistence, cloud-services-team
phaultfinder updated the task description for T380182: Inbound interface errors.
Fri, Nov 22, 5:04 PM · SRE, ops-eqiad, DC-Ops
Maintenance_bot added a project to T380473: Decommission parse20[01-20]: SRE.
Fri, Nov 22, 4:29 PM · SRE, DC-Ops, ops-codfw, decommission-hardware, serviceops
jcrespo closed T376892: Expand media backup storage available space to 960 TB per datacenter as Resolved.

This is now done. Catchup and purging is ongoing, but after that finishes, we should be able to store almost 1 PB of media backups on both datacenters. Backups will continue without problem or manual interventions in the next months.

Fri, Nov 22, 4:19 PM · Patch-For-Review, media-backups, Data-Persistence-Backup, SRE
jcrespo closed T376892: Expand media backup storage available space to 960 TB per datacenter, a subtask of T262668: WMF media storage must be adequately backed up, as Resolved.
Fri, Nov 22, 4:16 PM · media-backups, Data-Persistence-Backup, Epic, Goal, SRE, SRE-swift-storage
phaultfinder updated the task description for T379668: PDU sensor over limit.
Fri, Nov 22, 3:54 PM · SRE, DC-Ops, ops-eqiad
Stang moved T380109: Mailing list for zhwiki arbcom from Site configuration to Closed on the Chinese-Sites board.
Fri, Nov 22, 3:31 PM · SRE, Chinese-Sites, Wikimedia-Mailing-lists
Ladsgroup closed T380109: Mailing list for zhwiki arbcom as Resolved.

https://lists.wikimedia.org/postorius/lists/wikipedia-zh-arbcom.lists.wikimedia.org Creating wikis is a much bigger work, that'll take a while

Fri, Nov 22, 3:29 PM · SRE, Chinese-Sites, Wikimedia-Mailing-lists
phaultfinder updated the task description for T379668: PDU sensor over limit.
Fri, Nov 22, 3:09 PM · SRE, DC-Ops, ops-eqiad
MoritzMuehlenhoff added a comment to T376737: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged).

Ack, thanks. Either is fine with me, I can also switch them to insetup and then keep them running.

Fri, Nov 22, 2:54 PM · Patch-For-Review, SRE, Traffic, ops-magru
RobH added a comment to T376737: magru: Incorrect racking for magru hosts (F-25G and Custom Config interchanged).

We were discussing this last week, and brianstormed some on https://etherpad.wikimedia.org/p/magru_server_swaps from the irc output:

Fri, Nov 22, 2:31 PM · Patch-For-Review, SRE, Traffic, ops-magru
phaultfinder updated the task description for T379668: PDU sensor over limit.
Fri, Nov 22, 2:19 PM · SRE, DC-Ops, ops-eqiad
phaultfinder updated the task description for T380182: Inbound interface errors.
Fri, Nov 22, 12:59 PM · SRE, DC-Ops, ops-eqiad
gerritbot added a comment to T379343: Create bookworm-based build host.

Change #1094387 merged by Muehlenhoff:

[operations/puppet@production] package_builder: Cleanups

https://gerrit.wikimedia.org/r/1094387

Fri, Nov 22, 12:46 PM · Patch-For-Review, Infrastructure-Foundations, SRE
gerritbot added a comment to T379343: Create bookworm-based build host.

Change #1094393 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make docker::baseimages ensurable

https://gerrit.wikimedia.org/r/1094393

Fri, Nov 22, 12:25 PM · Patch-For-Review, Infrastructure-Foundations, SRE
jcrespo added a comment to T376892: Expand media backup storage available space to 960 TB per datacenter.

Timestamp is in CET:

[12:26:36] <jinxer-wm> RESOLVED: DiskSpace: Disk space backup2011:9100:/srv/objectstorage 5.998% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=backup2011 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace
Fri, Nov 22, 11:33 AM · Patch-For-Review, media-backups, Data-Persistence-Backup, SRE
Maintenance_bot added a project to T380564: decommission ganeti1011 / ganeti1014: SRE.
Fri, Nov 22, 11:29 AM · SRE, DC-Ops, ops-eqiad, decommission-hardware
Clement_Goubert closed T376966: wikikube-worker21[56-70] implementation tracking, a subtask of T376965: Q2:rack/setup/install wikikube-worker21[56-70], as Resolved.
Fri, Nov 22, 11:26 AM · SRE, serviceops, ops-codfw, DC-Ops
jcrespo added a comment to T376892: Expand media backup storage available space to 960 TB per datacenter.

Capacity reached 94.2% and finally it is on a downward trend: 93.7% 🎉

Fri, Nov 22, 11:22 AM · Patch-For-Review, media-backups, Data-Persistence-Backup, SRE
• ayounsi updated the task description for T380050: Decommission E/F 8 Dell switches.
Fri, Nov 22, 11:19 AM · Patch-For-Review, SRE, DC-Ops, ops-eqiad
gerritbot added a project to T379343: Create bookworm-based build host: Patch-For-Review.
Fri, Nov 22, 11:18 AM · Patch-For-Review, Infrastructure-Foundations, SRE
gerritbot added a comment to T379343: Create bookworm-based build host.

Change #1094387 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] package_builder: Cleanups

https://gerrit.wikimedia.org/r/1094387

Fri, Nov 22, 11:18 AM · Patch-For-Review, Infrastructure-Foundations, SRE
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits