• Rebuilt damned-near the whole lab on Windows Server 2025: WS2025 Hyper-V host, Debian guests, plus DC, CA, and fileserver VMs (all 2025).

• Homelabs are how you meet the newest version of the thing, and its rough edges, before a change window forces the introduction.

• The Take: “supported” is a claim a vendor makes, not a promise the software keeps; test the mainline path in the lab from time to time, not just the weird edge cases.

• Podcast episode 053: how to survive as a solo SysAdmin, with Eric Siron.

• Tool of the Week is Smallstep step-ca, an open-source internal CA that makes lab certificates a lot less painful than they were this month.

  👋 New here? Subscribe to SysAdmin Weekly for great IT content, and SysAdmin therapy.

From the Console

I spent the last few weeks tearing my lab down to the studs and rebuilding (again) on Windows Server 2025. Hyper-V now runs the virtualization layer, and the part that still makes me grin is what sits on top: a stack of Debian VMs, right alongside a domain controller, a certificate authority, and a file server, all three of those on Server 2025 as well. Windows hosting Linux guests alongside the AD estate. That is the lab for now.

This is the whole reason any of us run homelabs. Not the rack lights. It is the one place you get to meet the newest version of the thing on your own terms, before a customer or a change window makes the introduction for you. Staying current is not a nice-to-have in this job; it is the job.

What do the “what’s new in Server 2025” posts skip? Labbing the new stuff does not just show you the new capabilities. It shows you which of the vendor’s own instructions (and docs) quietly do not hold. I went to stand up Windows Admin Center on that fresh fleet, the documented and supported way, and lost most of an afternoon to it. The supported installer walked right up to the certificate step and silently failed to bind the cert to HTTPS (amongst many other issues), on a clean box, following the official docs to the letter. Four hours, and about ten minutes of that was genuinely my fault. I am writing the full autopsy up as a blog post for AndyOnTech.com, so I will not spoil the ending here. The short version is the thread running through this whole issue: “supported” is a claim a vendor makes, not a promise the software keeps, and the lab is where you find that out on your own time instead of in production.

The latest on the SysAdmin Weekly Podcast

Episode: How to Survive as a Solo SysAdmin: Where to Start When You’re the Only One Topic: When you are the only person between a working business and total collapse, the job stops being about doing everything and starts being about deciding what not to do this week.

Why SysAdmin’s should tune in:

• It is a survival guide for the team of one: triage before projects, documentation as insurance, and buying your time back with automation and the right managed services partner.

• Ignore it and you burn out, or you get hit by a bus and the business finds out the hard way that everything important lived in your head.

• For anyone who got hired into, downsized into, or promoted into owning the whole environment

• Plus a News React on Windows Server hot patching and a Nerd Hour on Debian 13.6 Secure Boot certs.

Watch on YouTube

Listen on Spotify

The Take

Vendors publish a “supported path” for a reason, and we lean on it for a reason. It is meant to be the route that has been tested, documented, and blessed, the one you can trust when you do not have time to reverse-engineer the product. So here is the uncomfortable thing I got reminded of this week. Supported is a claim, not a guarantee. The documented, supported way to do a routine thing with Windows Admin Center quietly failed multiple ways throughout the installation / setup, and nothing in the process warned me or provided meaningful error messages.

When you are the solo or lean-team admin like the theoretical solo SysAdmin we discussed in this week’s featured podcast episode, you plan your maintenance windows around the assumption that the supported path works. You do not budget four hours to install a management tool the vendor says installs in twenty minutes. The exotic edge cases you already distrust, so you test those. It is the boring, supported, “this obviously works” step that sometimes surprisingly wrecks your evening, precisely because you had no reason to give it a second look.

So test the happy path, not just the weird one. The install everyone says just works, the migration the docs call routine, the failover you assume is fine because the vendor says “It’s fine… trust me bro”. Run it once in the lab before you run it live. Not because you are paranoid, but because “supported” and “actually works in your environment” are two different sentences, and the gap between them is your outage. I would rather find that gap in my basement or test/dec environment than in a change window with someone waiting on me.

Community Signal

James van den Berg - “Windows Admin Center 2511 and the Security of Windows Server” - Van den Berg is a long-running Microsoft MVP for Cloud and Datacenter Management, and this is a clean walkthrough of using Windows Admin Center to drive the Server 2025 security surface: OSConfig baselines for CIS and DISA STIG, Secured-core, TPM attestation, and VBS. What earns it the slot this week is that he tells you to test it in a lab before you point it at production. That is the exact reason my own lab exists, and it is a reminder that WAC, on-ramp gripes aside, is a genuinely useful console once it is running.

Florian Burnel / IT-Connect - “Debian 13.6 Is Out: Here Are the Key Changes” - Burnel is a Microsoft MVP for Cloud and Datacenter Management, and his rundown of the Debian 13.6 point release is a near-perfect illustration of this week’s whole point. Microsoft’s old UEFI Secure Boot certificate authority, the one that has quietly signed bootloaders for well over a decade, expired in June 2026, and Debian 13.6 ships the fwupd 2.0.20 update that rotates affected machines onto the Microsoft UEFI CA 2023 certificate before a future bootloader update leaves a Secure Boot box unable to start. Nobody signed up for “my certificate expired, so my server will not boot,” and yet here we are. If you run Debian anywhere in the estate, read this before your next kernel or shim update makes the decision for you; it is the exact terrain Eric and I walk through at Nerd Hour this week.

Tool of the Week

Smallstep step-ca - An open-source, online certificate authority with a built-in private ACME server, so your internal boxes can automate certificate issuance and renewal the same way the public web does with Let’s Encrypt.

If you have ever wanted TLS everywhere in the lab without hand-rolling OpenSSL commands or standing up a full ADCS deployment, this is the lighter path. It speaks ACME, so certbot, Caddy, Traefik, nginx, and cert-manager all just work against it, and it issues short-lived certs with automated renewal so you are not babysitting expirations. Honest scope: it is Apache-licensed and actively maintained, and it is aimed at DevOps and homelab use, not at replacing an AD-integrated enterprise PKI. There is no OCSP or CRL and no Active Directory Certificate Services integration, so if you need GPO autoenrollment and AD-authenticated templates, that is still ADCS territory. One caution: it has patched a run of provisioner authorization CVEs over the past year, so run a current release (0.30.0 or newer) and do not expose the SCEP endpoint if you are not using it. For a lab, or for internal service certs on a private network, it removes a lot of pain.

Quick Win of the Week

Take the last thing that cost you half a day of troubleshooting and write the ten-minute version of it. Not a polished KB article; the terse “here is what was actually wrong and here is the command that fixed it” note that your future self, or the next admin, will be desperate for at 2am. Drop it wherever your hit-by-a-bus documentation lives. The four hours are already spent; the only question is whether anyone, including you, ever gets those hours back.

Fun Retro SysAdmin Fact

Windows Admin Center did not start life with that name. Microsoft unveiled it at Ignite in September 2017 under the codename “Project Honolulu”. I was there and remember it! Microsoft pitched as the browser-based console that would finally pull the scattered MMC snap-ins into one pane of glass, and it ran under that codename all through the preview. It did not pick up the Windows Admin Center name until it went generally available on April 12, 2018. So the tool that ate my afternoon this week is only about eight years into carrying its real name, and apparently still working a few of the install-day kinks out of its system.

Until Next Week

Run the boring supported step in the lab once before you trust it in production. Remember…. supported and working are not the same sentence.

Stay Frosty,

SysAdmin Weekly

• The most valuable skill I saw at a security conference this week had nothing to do with security: it was the ability to say what a technical control does in a sentence a CFO can repeat.

• Highlighting 052 of the podcast: Andy and Eric Siron dig into why homelab and server hardware went vertical in 2026, who is eating the supply, and why the prices probably do not come all the way back.

• The Take: hardware just got expensive in the worst budget climate in years, and “DDR5 is up 125%” is not a sentence that funds a refresh; the business-outcome version is.

• Community Signal: Packet Protector’s episode on translating security work into business outcomes, with concrete moves like reading your company’s own earnings call before you ask for money.

• Tool of the Week is Netdata, because the fastest way to win a hardware argument is to walk in with the utilization data instead of a vibe.

  👋 New here? Subscribe to SysAdmin Weekly for great IT content, and SysAdmin therapy.

From the Console

I spent this week at a security conference in the Dallas area, and the most useful thing I heard all week had nothing to do with security.

It was at XChange Security, a room full of MSPs and MSSPs who run other people’s infrastructure for a living. These are not people who need convincing that email authentication matters. And yet the conversation that kept surfacing, over and over again, was not about attack surface or detection coverage. It was about how to keep a security practice funded when the person signing the checks does not understand a single thing you do.

One thing I’ve learned after being in tech as long as I have…. No executive has EVER cared how your DMARC, DKIM, and SPF records are configured. That is a genuinely interesting conversation to have with me, at nerd hour, over a beer. Like…seriously. I’ll geek out over DMARC with all the other techs. That said, it’s a terrible conversation to have with your CFO. What the CFO wants to know is whether someone can impersonate the company to its customers, its vendors, and its bank. That is the whole question. Same three records, completely different sentence.

We are in a stretch where a lot of IT departments are under a microscope on both budget and headcount. In that climate, the ability to do the technical work is table stakes; it is assumed, and it does not save your seat. The thing that saves your seat is being the person who can stand in front of leadership and translate the work into an outcome they can act on. Not “we deployed the thing.” Instead: “we closed the gap that lets someone pretend to be us.”

Call it the translation layer. It sits between what you actually did and what the business heard, and most technical people never build it. The ones who do are the ones who are still employed when the spreadsheet gets tight.

The latest on the SysAdmin Weekly Podcast

Episode: Why Is Homelab Hardware So Expensive in 2026 (and When Will It Get Cheaper)?

Topic: Andy and co-host Eric Siron get into why memory, NAND, and storage prices went vertical in 2026, who is actually absorbing the supply, and what practitioners should do about a hardware market that stopped making sense.

Why it’s worth your time:

• It puts real numbers on the pain: a Raspberry Pi board that was $120 is now $305, write-hardened NVMe drives are priced into orbit or flat out of stock, and Andy admits to pricing used flash storage on eBay before he pumped the brakes and re-architected his whole lab.

• It explains the mechanics you will be quoting to your own boss soon: the AI buildout is eating memory and NAND capacity, three companies control roughly 95% of DRAM, a new fab takes three to five years, and Gartner has prices settling permanently above pre-shortage levels.

• It lands on the skill this crunch is quietly reviving: right-sizing hardware to the workload instead of reflexively throwing spec at the problem, which is the discipline the cheap-hardware era let atrophy.

Watch on YouTube

Listen on Spotify

The Take

Your next hardware refresh got a lot more expensive, and you are going to have to go ask someone for the money. In this budget climate….. Good luck.

The episode above explains why the prices are what they are, and the short version is bleak: this is structural, not a blip, and it may not fully reverse for years. That is the part you already feel every time you open a vendor quote. Now…what happens next is that you have to walk into a room and justify a bigger number than last year to someone who was already looking for reasons to cut. And the way most of us instinctively make that case is exactly the way that fails.

Here is the failure mode. You lead with the mechanics, because the mechanics are true and you find them compelling. “DDR5 is up 125%, NAND is worse, the drives I need are on three-month lead times, and the refresh I scoped at forty grand is now closer to sixty.” Every word of that is correct. None of it funds anything. The person across the table heard “prices went up and IT wants more money,” which is the least persuasive sentence in the building.

Now translate it. “The storage under our billing system is past warranty and out of spare capacity. If it fails, we cannot invoice customers until it is replaced, and replacement parts are now measured in months, not next-day. I want to refresh it while we can still choose the timing instead of having the timing chosen for us.” Same request. Same dollars. But now it is a business-continuity decision with a clock on it, not a line item that went up. That is a sentence a CFO can carry into their own meeting and repeat.

That is the whole game right now, and it is the same muscle from up top in a different room. The hardware crunch did not just make gear expensive. It made the budget conversation harder at the exact moment the budget conversation got more hostile. The admins who get their refreshes approved this year will not be the ones with the best spec sheets. They will be the ones who showed up with the utilization data and the business consequence, and left the part numbers in their back pocket unless someone asked.

Community Signal

Packet Protector - “Translating Security Objectives into Business Outcomes” - Hosts JJ Minella and Drew Conry-Murray sit down with Eyvonne Sharp, who spent twenty-plus years going from technical IC to leadership, and the whole thing is a practical manual for the translation layer I keep going on about. The moves are concrete and a little uncomfortable in a good way: read your own company’s earnings call or 10-K to find out what leadership actually cares about before you ask for a dime, quantify your work in revenue or mission terms instead of hours saved, and drop the fear-based “reputational damage” pitch because seasoned executives have heard it a hundred times and tuned it out. If The Take this week resonated, this is the long-form version from people who have sat on both sides of the table.

Tool of the Week

Netdata - a free, open source (GPLv3+) monitoring agent that gives you per-second CPU, memory, disk, and network metrics on every node, with real-time dashboards and zero configuration to get started.

I am putting it here for a specific reason tied to The Take: the fastest way to win a hardware argument is to stop arguing and show the graph. Netdata is how you find out that the database server everyone swears is “maxed out” is sitting at 11% memory utilization, or that the box you were about to leave alone is genuinely out of headroom. That is the difference between right-sizing on data and right-sizing on a hunch, and it is the receipt you bring to the budget conversation. Honest scope: the agent is free and open source and runs comfortably on a homelab node at around 5% of a core and 150MB of RAM, and it scales up to real infrastructure. The catch is retention and the single-pane view across many machines. Long local history and a unified dashboard across a fleet lean on Netdata Cloud, which has a usable free tier for individuals and paid tiers once you are a team, or on self-hosting parent nodes if you want to keep everything in-house. For a homelab or a small shop that just needs to see what is actually happening right now, the free agent alone earns its keep.

Quick Win of the Week

Take the last status update or upgrade request you sent up the chain, the real one, and rewrite the opening sentence so a non-technical executive could repeat it out loud without you in the room. Cut every acronym and every part number from that first line. If you cannot state the business outcome or the business risk in one sentence, that is not a writing problem, it is a sign you have not figured out why the work matters to anyone holding a budget, and better to learn that at your desk than in the meeting. Fifteen minutes, one sentence, and you will use the skill again within the month.

Fun Retro SysAdmin Fact

Hardware sticker shock is not new: in 1956, IBM’s 305 RAMAC, the first computer to ship with a moving-head hard disk, stored a whopping 5 million characters (about 5MB) across fifty 24-inch platters and leased for roughly $3,200 a month, which is north of $36,000 in today’s money, so the next time a NAND quote makes you wince, remember your homelab holds a few million times that for the price of a nice dinner.

Until Next Week

The technical work is the part you already know how to do; the sentence you say to the person holding the budget is the part that keeps you employed.

Stay Frosty,

Andy

SysAdmin Weekly

• Thunderbird 145 now speaks native Exchange (EWS with modern OAuth2), which means an open source mail client that finally works with Microsoft 365 without legacy auth or IMAP duct tape.

• New episode 051: Andy and returning guest Clay Tamam tour what actually runs in their labs, hardware to hypervisors to the Graveyard of gear they powered off.

• The Take: “end of life” is often a status you assigned, not a permanent fact; the tools you wrote off deserve a revisit cadence, because the reason you quit them sometimes gets fixed.

• Community Signal: Brandon Lee on the popular homelab advice he no longer follows, which is the same right-size-the-lab discipline the episode is built on.

• Tool of the Week is Jellyfin, the open source answer to the latest round of Plex price and paywall changes.

From the Console

I did not expect to feel anything about an email client in 2026. Then Thunderbird went and surprised me.

For years, Thunderbird was effectively dead to me for anything touching Microsoft 365. If you wanted it to talk to Exchange Online, you were bolting on a paid add-on or falling back to IMAP and legacy authentication, which is to say the exact protocols Microsoft has spent years trying to kill off, and for good security reasons. So on my Debian box, the machine I actually enjoy working on, I had no real open source mail client for my M365 mail. I lived in a browser tab and told myself that was fine.

Version 145 changed that. Thunderbird now speaks Exchange Web Services natively, uses the standard Microsoft OAuth2 sign-in, and detects the account settings for you. No add-on. No legacy auth. No pretending IMAP is a plan. I set it up, it just worked… mostly (after I accounted for bug 1847846 due to my large mailbox size). After that, I sat there mildly stunned that a piece of software I had mentally filed under “lost cause” had quietly clawed its way back into my daily driver rotation.

We wrote a whole issue about this once. Issue #22 was about open source as a lifeline: the exit door you keep unlocked for when a vendor changes the terms on you. This is the other half of that same coin. Sometimes the lifeline is not a new tool at all; it is one you already knew, left for dead, that got picked back up and reworked to fit the modern world. More on that below.

The latest on the SysAdmin Weekly Podcast

Episode: What I Actually Run in My Homelab (and Why)

Topic: Andy is joined by returning guest Clay Tamam, a working SysAdmin in the Netherlands, for an honest tour of what is really sitting in their labs: the hardware, the hypervisors, the services they use every day, and the reasoning behind each choice.

Why it’s worth your time:

• It is the antidote to homelab-flex culture: two practitioners talking about what earns its place and what got powered off, not what looks impressive in a rack photo.

• Andy walks through tearing a four-node Kubernetes cluster down to five VMs on a single Debian box, and Clay builds a rack from scratch on a real budget, so you get both ends of the spectrum with the tradeoffs spelled out.

• It closes with the Graveyard: the gear and services that got shut off and why pruning is part of the discipline, not a failure of it.

Watch on YouTube

Listen on Spotify

The Take

“End of life” is a status you assign as often as one a vendor declares. And the two are not the same thing.

The podcast episode (above) closes on the Graveyard, the pile of gear and services you power off because they stopped earning their place. That discipline is correct. Most labs, and most production stacks, are carrying dead weight that nobody has had the nerve to switch off. Don’t get me wrong, removing technical debt is a good thing (in most cases) but the graveyard is not a one-way door. Sometimes a tool you buried was not actually dead. It was just dead to you, because the specific thing that made it unusable had not been fixed yet.

Thunderbird is my case in point this week. I wrote it off for M365 work years ago, for a completely valid reason at the time, and then I stopped tracking it. That is the trap. “I wrote it off” quietly turns into “I stopped paying attention,” and the two feel identical right up until the day the project ships the exact fix you gave up waiting for. Thunderbird 145 added native Exchange support over EWS with modern authentication, and the coverage was blunt about how long people had wanted it. The reason I quit it got fixed. I just was not looking.

Let’s be specific about the discipline, though, because this is not nostalgia. A tool earns its way back on current merits, not sentiment. Is it actively maintained? Does it meet your real requirements today? What is the catch? Thunderbird bet on EWS, a protocol Microsoft is slowly phasing out in favor of Graph, so the honest read is that email works now, calendar and contacts are still in progress, and the team is already building toward Graph for when EWS finally sunsets. That is a tool worth adopting with eyes open, not a fairy tale. The point is the cadence: put the tools you wrote off on a revisit schedule instead of treating a one-time verdict as a life sentence. The graveyard occasionally coughs one back up, and you want to be the admin who noticed.

Community Signal

Brandon Lee - “I No Longer Follow This Popular Home Lab Advice” - Brandon has decades in the field, and this is the same right-size-your-lab thesis Episode 051 of the podcast (with a mention in 052!) is built on. He makes the case for using Kubernetes only when orchestration actually solves a problem, applying high availability selectively instead of everywhere, and treating a couple of services (email and password managers among them) as reasonable exceptions where managed reliability beats self-hosting. It is a rare homelab piece that argues for running less, and it pairs directly with my Kubernetes-to-VMs teardown in episode 051.

Tool of the Week

Jellyfin - a fully open source, self-hosted media server for your movies, shows, and music, with no accounts to create, no cloud dependency, and no feature you rely on sitting one pricing update away from a paywall.

The reason it earns the slot this week: Plex just tripled the price of a new lifetime Plex Pass to $749.99, effective July 1, on top of already gating remote streaming behind a paid pass, which is exactly the vendor-changes-the-terms moment that sends people looking for the exit. Jellyfin is that exit, and it is genuinely free with no premium tier gating the core experience. Honest scope, though. This is not a turnkey Plex clone. Hardware transcoding takes real configuration to get right, the client apps across smart TVs and streaming boxes are less polished than Plex’s, and Jellyfin ships no built-in remote-access magic, so getting to your library from outside the house means you bring your own reverse proxy or a mesh VPN like Tailscale (which came up in the episode for exactly this reason). For a homelab that already runs a hypervisor and a bit of networking, that is a comfortable weekend project. For a non-technical household that just wants to press play, Plex is still the smoother ride, and it is fair to say so.

Quick Win of the Week

Pick one tool you wrote off more than two years ago, one you filed under “dead” and stopped tracking, and spend thirty minutes with its current changelog and release notes. Not to adopt it. Just to find out whether the specific thing that made you quit it has been fixed. Thunderbird and Microsoft 365 was mine this week; you almost certainly have your own candidate sitting in your mental graveyard. The goal is to make “I wrote it off” a decision you revisit on a schedule, not a verdict you handed down once and never appealed.

Fun Retro SysAdmin Fact

Thunderbird 1.0 shipped on December 7, 2004, which makes Mozilla’s mail client older than most of the “modern” SaaS platforms built to replace it; it was very nearly left to rot when Mozilla stepped back from active development in 2012, and here it is twenty-one years later adding native Exchange support and quietly having a renaissance.

From My Other Corner of the Internet

AndyOnTech

The reconstruct-project-history piece I had been sitting on is finally live: Can Claude Code Reconstruct a Project History You Never Documented?. It walks through pointing Claude Code at nothing but a repo’s git history and having it rebuild the changelog and decision trail that no human ever sat down to write. If you have a months-old project you no longer fully understand and you’re asking yourself the question “how did I get here?”, this is the practical, boring-value use case for the tooling, not the hype one.

Until Next Week

Prune what stopped earning its place, but keep a list of the tools you buried; a few of them are going to knock.

Stay Frosty,

Andy

SysAdmin Weekly

• Right tool for the right job is a discipline, not a budget line: stop sizing your tech to your ego and start sizing it to the task.

• New episode 050: how to run a blameless incident postmortem that ends with a root cause instead of a name, whether you’re a war room of sixty or a shop of one.

• The Take: your own postmortem is half the skill; reading everyone else’s, including your vendors’, is the other half, and their transparency is a procurement signal.

• Tool of the Week is OneUptime, an open-source stack that captures the incident timeline automatically so your postmortem has real material to work from.

• Quick Win: steal the show’s free postmortem template and fill in the boilerplate now, before the pager goes off.

From the Console

One of the most expensive habits in tech has LONG been reaching for the biggest thing on the shelf just because it happens to be sitting there.

I caught myself doing it last week. I had a two-paragraph text file that needed summarizing, and my hand instinctively went straight for the largest, most capable AI model I had access to (because i’ve been doing a lot of complex work lately). Had I not caught myself, that’d be like renting a semi to move a bag of groceries. Opus does not need to summarize your grep output. Haiku is not going to architect your app. The skill is not knowing which model is “best,” it’s knowing which one fits the job in front of you, and most jobs are small.

The lab version of this is worse, because the lab version has blinking lights. Nobody needs a half-rack SAN to hold a few text files and a 2GB media library, and yet the homelab channels are full of gear that exists mostly because it was available and the owner could. I have been guilty of this too. Right tool for the right job is not a spending rule, it’s a discipline and arguably MORE important in today’s world of insane hardware prices that it used to be. The moment you size the tech to the task instead of to your ego, everything gets cheaper, quieter, and easier to reason about.

The latest on the SysAdmin Weekly Podcast

Episode: How Do You Run a Blameless Incident Postmortem?

Topic: Andy and Eric Siron pull from combined decades of incident reviews to break down what a postmortem actually is, what kind of outage earns one, who belongs in the room, and why the follow-through is the whole point.

Why this one matters:

• An effective postmortem turns an outage into tangible fixes instead of finger-pointing, which is the difference between an incident you learn from and one you repeat.

• Skip it and the same failure comes back, because nobody wrote down the contributing factors and nobody owned the action items.

• It’s for anyone who owns uptime (which is MOST SysAdmins), and the discipline scales: sixty people in a war room or just you writing a summary for one nervous boss.

Watch on YouTube

Listen on Spotify

The Take

Running your own postmortem is half the skill. Reading everyone else’s is the other half, and almost nobody treats it that way.

Every vendor you depend on has a track record of how they behave when they break. Some of them tell you. In February, Clerk put out a postmortem that named the actual root cause: a Postgres auto-analyze flipped a query plan because the planner’s sample size was too low, the bad plan strangled the database, and roughly ninety minutes later a manual re-analyze brought it back. That is a real root cause, with the communication failures owned in the same document. You read it and you know exactly what happened and what they changed.

Now, that willingness to publish a specific cause is itself a signal, and it correlates almost perfectly with how a vendor treats you at 2am. When GitHub’s own engineering leadership publicly attributes a run of outages to tightly coupled architecture that let local problems cascade, and to systems that couldn’t shed load from misbehaving clients, that is a company telling you how it actually failed. Compare that to the outfits whose entire public record is “we experienced elevated error rates” followed by silence. One of those is a partner. The other is a liability you haven’t been billed for yet.

So make it part of procurement. Before you take a hard dependency, go read the vendor’s status history and their last two or three postmortems. Not the marketing uptime number, the writeups. If they don’t publish any, or if every one is corporate mush with no named cause, you already know how the next outage conversation goes. The postmortem discipline Episode 050 is about does not stop at your own perimeter.

Community Signal

Awesome community work worth your attention.

Lorin Hochstein - “Quick thoughts on GitHub CTO’s post on availability” - Hochstein is a resilience-engineering practitioner (ex-Netflix SRE) who writes some of the clearest incident analysis on the internet, and here he picks apart GitHub’s own writeup on its recent outages. It’s the perfect companion to this issue’s Take: this is what it looks like when a working engineer reads someone else’s postmortem and pulls the real lessons out of it.

Google SRE - “Postmortem Culture: Learning from Failure” - Chapter 15 of the free SRE book, and still the clearest articulation of blameless-not-unaccountable in print. If Episode 050 had you nodding, this is the reference to hand the skeptic on your team who thinks “blameless” means “nobody is responsible.”

Tool of the Week

OneUptime - an open-source, self-hostable reliability stack that rolls monitoring, on-call scheduling, status pages, and blameless postmortems into one platform.

The reason it earns the slot this week: it captures the incident timeline automatically while the fire is still burning, which is exactly the raw material a good postmortem needs and exactly what nobody remembers to collect by hand at 2am. Honest scope, though. This is a full platform, not a single container you set and forget. Self-hosting it is real infrastructure with a database and the works, so for a homelab it’s a weekend project, not a five-minute one; if you just want uptime checks, Uptime Kuma is the lighter tool. And to be straight with you: this is a smaller, less grassroots-proven community than a darling like Uptime Kuma, and it’s an all-in-one bet rather than a best-of-breed one. What earns it the slot is that it’s genuinely fully open on GitHub (Apache 2.0, not open-core), actively maintained, and its postmortem tooling actually exists instead of being bolted on.

Quick Win of the Week

Drop a real postmortem template into your team wiki or repo today, before you need it. We linked a markdown postmortem template that i’ve used over the years in the resources section of the last podcast episode. That version is free, CC BY 4.0, and version-tracked in the SysAdmin Weekly community repo here.

Pre-fill the parts that never change (your severity levels, escalation contacts, and the timeline table headers) so that when something does break, you’re filling in facts instead of formatting a document while the help-desk phone screams.

Fun Retro SysAdmin Fact

On January 15, 1990, AT&T’s long-distance network collapsed for about nine hours and blocked roughly half of all calls placed through it, an estimated $60 million in lost connections, and the root cause was a single misplaced break statement in the C recovery code running on its 4ESS switches: proof, 36 years early, that “blameless” exists because the cause is almost never a person.

Question I Got Asked (and the Real Answer)

The question: “I’m a one-person shop. Do postmortems even apply to me?”

The common (wrong) answer: “No, that’s enterprise process overhead. You already know what broke.”

The real answer: The document scales down to a paragraph; the discipline does not. You will not remember, three months from now, the exact order you tried things at 2am or why the obvious fix didn’t work. A five-minute writeup with a timeline, a root cause, and two action items you actually close is worth more to a solo admin than a forty-page template is to a team that files it and forgets it. Blameless is easy when the only name in the room is yours. The hard part, same as everywhere, is the follow-through.

Until Next Week

Size the tool to the task, write down what broke, and close the action item you keep meaning to close.

Stay Frosty,

Andy

SysAdmin Weekly

• After a few issues of me telling you not to trust the machine, here is the other half of the ledger: the tangible, boring AI wins that are actually worth your time.

• Episode 049 is out: how attackers run local LLMs on their own hardware to generate spear phishing at scale, why “spot the typo” training is dead, and where the real defensive line sits, with Eric Siron.

• The Take: the same accessible AI that attackers point at scaling the bad stuff can be pointed at the valuable, well-scoped work you never had hours for, including understanding the threat landscape itself.

• I rebuilt a four-month-old project’s entire changelog from nothing but the git diffs in about twenty minutes. The scoping that made it cheap is the same scoping that made it good.

• Tool of the Week: Fabric, reusable AI “patterns” you run from the command line against hosted or local models.

From the Console

I have spent the last several issues being the guy telling you not to trust the machine. Doom Discounted, The Off-Switch You Don’t Own, the whole local-inference drum. I stand by every word of it. But that is one half of the ledger, and this week I want to read you the other half.

Because here is the thing I do not say enough when I am busy being skeptical: lately the boring AI wins have been quietly piling up, and they are real. The one that got me this week…. I pointed Claude Code at a four-month-old project of mine that had grown from a single throwaway script into a full workflow, and asked it to reconstruct the changelog I never bothered to keep, from nothing but the git diffs. About twenty minutes later I had an accurate, dated history of a project I could no longer fully trace in my own head. Not magic. Just a tedious, genuinely useful job that was never worth a human afternoon, suddenly worth twenty minutes.

That is the whole theme this week. Not “AI is amazing.” The lesson this week is that the floor on boring-but-valuable work just dropped, and we should go use that…..

And now, back to our regularly scheduled programming.

The latest on the SysAdmin Weekly Podcast

Episode: 049 - How Do Attackers Use Local LLMs to Phish at Scale?

Topic: Andy brings his InfoSecurity Europe session to the show, and Eric Siron joins to walk through how threat actors run open-weight models on their own hardware to generate targeted spear phishing at scale, in any language, with no internet and no guardrails, and where the real defensive line actually sits.

Why this one matters:

• The guardrail gap, made concrete: ask a hosted model to write a phishing lure and it refuses; pull the right open-weight model onto a laptop and that refusal layer simply is not there. The demo is several tailored spear-phishing lures in ninety seconds.

• Why “spot the typo” awareness training is dead, and why verification culture plus strong email authentication (SPF, DKIM, DMARC) is what actually carries the load now

• Plus the nerd hour and news react: Washington pumping the brakes on Fable, the New York ghost-gun-printing law, and the Google AI liability question

Listen on Spotify

Watch on YouTube

The Take

This week’s episode (above) is about attackers pointing cheap, ungoverned AI at the one thing they care about: scaling the bad stuff. Fifteen tailored spear-phishing lures in ninety seconds, no guardrails, no internet connection required. That is the threat, and it is real. Remember though, that the exact same accessibility cuts the other way, and most of us are barely using it.

Earlier this month I built a small tool that reads the places SysAdmins actually congregate, the subreddits, the forums, the threads nobody has time to keep up with, and correlates what people keep struggling with. Not vibes. The recurring, in-their-own-words pain that gets buried under a thousand individual posts. I built it to find things worth covering on the show. It turned into something more useful than that: a standing read on what is actually breaking in the field, and an early-warning lens on how the ground is shifting under us.

That second part matters more after recording Episode 049. The same engine that surfaces “everyone is fighting the same Intune policy bug this month” will also surface “attackers are openly comparing which local model writes the cleanest lures.” Pointing AI at understanding the field, including how the bad actors are using it against you, is not a gimmick. It is one of the highest-value things a defensive team can do with this technology right now, and A LOT of orgs aren’t doing it.

Here is the part that ties the changelog and the community tool together, and it is the whole game: both worked because I scoped them hard. Tell the model exactly what to look at, exactly what to ignore, and exactly what good output looks like, and it earns its keep. Hand it a vague “go figure out my industry” and you get expensive mush. The tangible AI wins are not the demos. They are the narrow, well-scoped, deeply boring jobs you never had the hours for. If you want the longer version of that argument grounded in day-to-day IT work, I wrote it up here.

Community Signal

Simon Willison - “Large Language Models can run tools in your terminal with LLM 0.26” - If The Take is the argument that well-scoped AI is the real win, this is the reference implementation of that idea from a credible independent voice in the space. Willison’s llm is the opposite of the chat-window-you-babysit: a command-line tool that pipes input in, logs every prompt to a queryable SQLite database, and now hands models controlled access to tools you define as plain functions, against hosted models or local ones through Ollama. The post is about a year old now, but it is the cleanest articulation of the Unix-philosophy approach to AI that the GUI hype keeps drowning out, and it pairs directly with Fabric below.

Tool of the Week

Fabric - an open-source framework that turns AI into a set of reusable command-line “patterns” you pipe text through, instead of a chat window you have to babysit.

Fabric is the tool that makes the “scope it hard” lesson above repeatable. Each pattern is a saved, version-controlled prompt for exactly one job: summarize this, extract the action items, analyze this log, and you run it from the terminal with Unix pipes like any other command. It talks to hosted models (Claude, GPT, Gemini) and to local ones through Ollama, so sensitive input can stay on hardware you own. Honest scope: it is genuinely useful for a solo practitioner or a small shop today, but it is not an enterprise platform with RBAC and audit trails, so treat it as power tooling for you, not a sanctioned production service for the org. MIT licensed, north of 40,000 stars, and actively developed as of this month.

Quick Win of the Week

Pick one repository you actually rely on that has no changelog, or one where you started the changelog too late, and have an AI backfill its history from the git diffs. The trick is entirely in the scope. Hand it the short list of files that define how the thing works, the modules, the pipeline definitions, the runbooks, not every variable tweak, and tell it to read only the diffs touching those paths. Tell it to leave any existing entries untouched, and give it the exact output format you want. Then read the result against the diffs before you trust it, because the history is the authority and the summary is not. Twenty minutes, and a project nobody on the team could explain end to end suddenly has a paper trail.

Fun Retro SysAdmin Fact

The AI that fools people is not new; it is sixty years old. In 1966, MIT’s Joseph Weizenbaum wrote ELIZA, a simple pattern-matching script that imitated a therapist by turning your own statements back into questions, and it was so convincing that his secretary, who knew exactly how trivial the program was, reportedly asked him to leave the room so she could talk to it in private. Weizenbaum spent much of the rest of his career unsettled by how readily people projected real understanding onto a machine that had none, a reaction we now call the ELIZA effect.

Source: Smithsonian Magazine - “Why Joseph Weizenbaum, Creator of the World’s First Chatbot, Turned Against AI”

Until Next Week

The skeptic in me is not going anywhere, but it’s important to state that the boring wins are real, and the only way you will believe that is to point the machine at one tedious, valuable job this week and watch it work.

Stay Frosty,

Andy

SysAdmin Weekly

· A frontier AI model can be switched off worldwide in 72 hours, and not by you. Anthropic’s Fable 5 and Mythos 5 proved it this month.

· Clearly it’s not a Claude problem. It is a dependency problem, and it is the strongest case yet for owning the hardware your inference runs on.

· SysAdmin Weekly Podcast Episode 048 is out: the AI doom narrative held up against the actual data with Eric Siron.

· The WEF Future of Jobs numbers say net +78 million jobs by 2030, and the fastest-growing skill categories behind AI and big data are networks and cybersecurity. The work that keeps infrastructure running is on the growth side.

· Tool of the Week: Ollama, the second time around. Same recommendation as Issue 24, sharper reason to act on it now.

From the Console

I stripped my lab down a few weeks ago. Pulled the always-on gear I was not really using, consolidated a couple of roles onto hardware I trust to keep running, and rebuilt the rest with longevity in mind instead of raw horsepower. I expected to give something up for that.

Turns out…..I have not, in fact, given up something for those changes…. Everything I actually depend on is running as well as it ever did, and my power bill came back noticeably lighter. Turns out a lot of what I was running was running because it was already on, not because I needed it on.

Given the recent news about Anthropic… it makes me thing about something. Every box in that pared-down lab is one I control. It does not get a worse model pushed to it overnight. It does not get switched off because of a directive passed down from three companies upstream of me (or the US Government for that matter…). Those cases sounded like a philosophical point a month ago. This week it stopped being philosophical.

And now, back to our regularly scheduled programming.

The latest on the SysAdmin Weekly Podcast

Episode: 048 - The AI Doom Narrative vs the Data: Layoffs, Energy, and Jobs in 2026 Topic: Andy and Eric Siron take the AI fear stories one at a time, the layoffs, the data center energy and water panic, the “office workers are gone in eighteen months” predictions, and put each one next to what the actual data says.

Why this one matters:

· The WEF Future of Jobs numbers, the post-COVID overhiring correction, and the hyperscaler nuclear and cooling commitments, read against the headlines instead of in place of them

· What AI actually looks like at a SysAdmin’s keyboard versus what the press would have you believe it looks like

· Plus the nerd hour: Linus Torvalds on AI-generated bug reports drowning the kernel security list, Edge caught storing passwords in clear text, and a Forgejo and Restic lab cleanup

Listen on Spotify

Watch on YouTube

The Take

A model you build your work around can be turned off in three days, worldwide, and you will not be the one who decides it.

That is not a hypothetical. Anthropic released Claude Fable 5 to the public on June 9, with the wider Mythos 5 class going to a narrow set of partners the same week. By the evening of June 12, both were dark. Not deprecated, not rate-limited; switched off for every customer on the planet. The trigger was a US export-control directive barring access by any foreign national, and because nationality cannot be filtered on a live API in real time, the only way to comply was to pull the models for everyone. Anthropic’s own foreign-born staff included.

The part relevant for us mere-mortal SysAdmins? None of that had anything to do with how we were using the model. You could have had a clean, paid, well-behaved workflow built on it, and it would still have gone dark on Friday night because of a conversation between an executive and a cabinet secretary that you were not in.

And that conversation is its own story. The reporting says Amazon CEO Andy Jassy is the one who flagged the concern to Treasury Secretary Scott Bessent, citing Amazon researchers who got Fable 5 to cough up cyberattack-useful output. Amazon is Anthropic’s investor. Amazon is Anthropic’s cloud host. And Amazon, this week, is the party that walked a competitor’s three-day-old model release into a regulator. I am not going to tell you what to read into that. I am going to tell you those three roles sitting in one company is exactly the kind of thing you want to know about when it comes to the supply chain under your tooling.

To be fair to Anthropic, they pushed back on the severity, noting the same capabilities are already reachable through other public models, and Opus 4.8 and Sonnet kept running the whole time. So this is not me telling you Claude is the problem. Claude is not the problem. The dependency is the problem.

We have been building this case in different shapes for a while. Issue 21 was about compute consolidating into a handful of hands. Issue 25 was about the hardware squeeze landing on your doorstep. Both of those were about cost and supply. This adds a third axis, and it is the sharpest one: availability you do not govern. A model running on hardware you own cannot be revoked by a directive you never saw. That is the whole pitch for local inference, and it just got handed the cleanest example it has ever had. The full timeline is here if you want to read it cold.

None of this means rip the hosted models out of your stack. They are still better than what most of us can run at home, and that gap is real. It means know which of your workflows would simply stop if a model went dark on a Friday, and have an answer that does not require anyone’s permission to switch on.

Question I Got Asked (and the Real Answer)

The question: “Be honest, is AI going to delete my job?”

The common answer: Some executive’s eighteen-month countdown, repeated back with the serial numbers filed off. Half the room nodding, the other half quietly updating their resume.

The real answer: The most-cited primary source on this is the WEF Future of Jobs Report, and the people repeating the doom number rarely seem to have read it. The report projects 170 million jobs created and 92 million displaced by 2030, for a net gain of roughly 78 million. That is real churn, about 22% of the jobs studied, and it is going to HURT in specific places; clerical and administrative roles, cashiers, and data entry are on the wrong side of it.

The more useful number in that same report is the skills outlook: the fastest-growing skill categories through 2030 are AI and big data, networks and cybersecurity, and technological literacy. Those are not the skills of a job being deleted. Those are the skills of the work that keeps the rest of it running.

The displacement is real and worth taking seriously. The version where it deletes the people who keep systems running is not what the data says. Again, Eric Siron and I spend a full episode of the podcast (above) on this distinction, and it is worth the listen, because the gap between the headline and the report is wide enough to drive a career decision through.

Community Signal

James Kilby - “Self-Hosting an AI Stack Using vSphere, Docker and NVIDIA GPU” - The practical version of everything above. Kilby walks through standing up a containerized inference stack on a single NVIDIA Tesla P4 inside vSphere, with the install commands and, the part I appreciated, the real power numbers: roughly 7 watts idle, 50 to 60 under active inference. If you read From the Console and wondered what “own your inference” actually costs to run, this is a concrete answer from someone who measured it. Updated June 2026.

Tool of the Week

Ollama - Open-source local LLM runtime. Pull a model, run it on hardware you own, expose an OpenAI-compatible endpoint any existing tool can talk to.

If you read Issue 24 you have already heard this pitch. The reason it earns the slot a second time is that the argument for it just got sharper. For issues 24, the case was privacy, cost, and not being told you cannot use AI at all. The Fable 5 takedown adds the axis I did not have a clean example for in May: availability you govern. Again…. a model running on hardware you own cannot be pulled by a directive you never saw.

Honest scope has not changed since Issue 24. Local model quality at consumer hardware tiers still trails the frontier hosted models, and for greenfield engineering work you will still reach for the hosted option more often than not. For the workflows that actually need to keep running no matter what happens upstream, this is the fallback The Take is asking you to have. The Quick Win below is “stand it up and prove it works.” An hour of your time, a permission slip from nobody.

Quick Win of the Week

Make a one-page list of every tool and workflow in your shop that calls a hosted AI model, and mark which ones would simply stop if that model went dark on a Friday night. For the one that matters most, pull an open-weight model in Ollama, point the tool at the local OpenAI-compatible endpoint, and confirm it actually runs. You are not switching off the cloud. You are proving you have a fallback that needs nobody’s permission to turn on, before the week you actually need it.

Fun Retro SysAdmin Fact

This is not the first time Washington export-controlled software. Through the 1990s, strong encryption was classified as a munition under the same export rules as weapons, and PGP author Phil Zimmermann spent roughly three years under federal criminal investigation for “exporting munitions without a license” after his code spread overseas; the case was dropped in 1996, and the idea that publishing software could be an arms-export crime has felt absurd ever since. Felt.

Until Next Week

Know which of your systems would keep running if the company upstream of you had a bad Friday.

Stay Frosty,

Andy

SysAdmin Weekly

• The compute consolidation I flagged back in Issue 21 isn’t a forecast anymore; it’s sitting in your parts bin.

• DRAM contract prices jumped roughly 90% in a single quarter, consumer SSDs climbed 147%, and a 16GB Raspberry Pi 5 that launched at $120 now runs $305.

• The hyperscalers didn’t just rent you the cloud; they bought up the fab capacity that makes the parts you’d use to avoid it.

• We pulled an older episode of the podcast to highlight this week > Episode 023 with Eric Siron from the vault: building a real lab without selling a kidney, more relevant now than the day we recorded it.

• This week’s moves: pool mismatched disks with SnapRAID, and actually measure how much write life your NVMe has left.

From the Console

I went looking for a couple of high-endurance NVMe drives last week. Not the bargain-bin kind; the write-tolerant drives you put under a workload that actually does sustained writes. I didn’t find them at a good price. I didn’t find them at a bad price either. They simply weren’t there.

Same story when I priced out more RAM for an existing box. Same story again when I looked at what a Raspberry Pi costs now versus a year ago. Somewhere in the last few months the math quietly flipped: buying new capacity stopped being a line item and turned into a project with a budget meeting attached.

So I’ve stopped trying to buy my way forward. I’m re-architecting what I already own, with longevity as the goal instead of raw horsepower. Make the gear I have last, instead of betting on a parts market that clearly isn’t betting on me.

This is a rant you’re going to be hearing about on SysAdmin Weekly for….. a while… I apologize in advance. =D

And now, back to our regularly scheduled programming…..

From the SysAdmin Weekly Vault

Instead of reaching for the latest episode of the SysAdmin Weekly podcast, we’re reaching for an older episode that fits with this week’s theme around PC hardware and homelabs.

Episode: SysAdmin Weekly 023 - Budget Home Lab Setup for IT Pros (with Eric Siron) Topic: Building a genuinely useful home lab without torching your budget, across five segments: hardware, software, virtualization, networking, and cloud.

Why this one still holds up, and matters more now:

• The hardware advice was always practical: don’t underestimate an old desktop or laptop with enough RAM, and lean on mini PCs as the modern budget chassis. For example, Eric stood up his entire lab on four 32GB nodes for around $4,000 (When we recorded that episode) The underlying logic used? > “cutting edge, not bleeding edge” (skip DDR5, stick with DDR4), reads less like budget tinkering and more like a 2026 survival guide today.

• Eric’s rule of thumb, brick a $100 box, not a $20,000 one, was about learning safely on cheap hardware. In a market where you can no longer reliably buy the $100 box either, it lands as an argument for defending what you already own and being patient with the gear that still works.

• The virtualization tour is still the cheapest path off the VMware tax: Hyper-V on Windows Server’s 180-day evaluation (re-arm it three times for roughly two years of runway), Proxmox if you miss the vSphere feel, libvirt and KVM for the Linux native route. Containers via Docker, Podman, or a small Kubernetes cluster cover the workloads where full VMs are overkill.

• Bonus reality check we did not plan for: Eric flagged in the recording that VMware trial licenses are now gated behind enterprise customer designation. If you ever wondered why I started talking about “the VMware tax” as a thing, that gate is where it started.

The Take

Back in Issue 21, I ran a thought experiment and landed on an uncomfortable answer: the consolidation of compute into a handful of hyperscalers would, eventually, take choice and independent operation away from the rest of us. I framed it as a question and said I thought the answer was yes. I did not expect the proof to arrive this fast, and I did not expect it to come through the supply chain instead of through policy.

So the ugly truth….. The three companies that make the world’s memory have pivoted their fab capacity toward high-bandwidth memory and enterprise parts for AI buildouts, because that’s where the margin is…. it’s simple capitalism. AI is on track to consume roughly a fifth of all DRAM production this year, per TrendForce. The downstream effect isn’t subtle: DRAM contract prices jumped around 90% in a single quarter, consumer SSDs climbed 147%, a 30TB enterprise SSD went from about $3,000 to $17,500, and that 16GB Raspberry Pi 5 went from $120 at launch to $305.

That’s the consolidation thesis, except it isn’t an abstraction about cloud strategy anymore. It’s the reason you can’t buy a stick of RAM or a decent NVMe drive at a price that makes sense. The same demand that’s building hyperscale AI capacity is the demand that’s pricing you out of building anything for yourself.

And that’s the quiet move now……. When the entities building the cloud also control who gets the silicon to build off the cloud, “just rent compute from us” stops being a sales pitch and becomes the path of least resistance, because every other path got more expensive on purpose. I’m not telling you to panic. I’m telling you to notice, and to make the gear you already own last, because owning your own compute is starting to look less like a preference and more like a position you have to defend.

Community Signal

One item this week, and fits right in with these week’s discussion.

Jeff Geerling - “DRAM pricing is killing the hobbyist SBC market” - Geerling lays out how LPDDR memory now makes up the majority of a single-board computer’s cost, which is why anything past 4GB of RAM has drifted out of hobbyist reach. His read lines up with mine: the theme of 2026 is repurposing what you already have, not buying new. If you want the hardware-level receipts behind everything I argued above, this is the post to read.

Tool of the Week

SnapRAID - parity-based protection for an array of ordinary, mismatched disks, which is exactly what you end up with when you can’t buy a matched set of new drives.

Full disclosure before I describe it: I haven’t run this one in production myself yet. It’s on the shortlist for the re-architecting work I’m doing now, and I’m flagging it here because it’s the community-tested answer (Perfect Media Server, r/DataHoarder, the Self-Hosted podcast crowd) for exactly the mismatched-disk world we’re heading into. I want it on your radar before you need it, not after.

SnapRAID computes parity across drives of different sizes and ages, so you can add whatever disk you scrounged this month and still survive a failure or two (it scales up to six parity disks if you want the headroom). Pair it with mergerfs and those odd-sized disks present as a single pool. Honest scope: this is snapshot-style parity synced on a schedule, not real-time RAID, so it’s built for data that doesn’t change by the second; media, backups, and archives are the sweet spot. Don’t drop a busy database or a hot VM datastore on it and expect RAID behavior. For bulk storage you want to keep alive for years on hardware you already own, it’s hard to beat. It also pairs cleanly with the eBay-and-government-surplus path Eric recommended on the Vault episode, since the whole point of that path is ending up with a stack of drives nothing else can pool.

Quick Win of the Week

Find out how much life your SSDs and NVMe drives actually have left, before the market forces the decision for you. On Linux, install smartmontools and run sudo smartctl -a /dev/nvme0 (swap in your device); read Percentage Used and Data Units Written for NVMe, or the Wear_Leveling_Count and Media_Wearout_Indicator attributes on SATA SSDs. On Windows, CrystalDiskInfo shows the same health and total-bytes-written data in a couple of clicks. Five minutes per box tells you which drives have years left and which ones you should quietly start planning around while you still can.

Fun Retro SysAdmin Fact

The RAM in 1960s mainframes was literally woven by hand: tiny magnetic ferrite rings threaded onto crisscrossing wires, often assembled by women hired for the fine motor skills the work demanded, and it’s why we still say a crashed program wrote a “core dump” and why some old-timers still call memory “core.”

Until Next Week

The cheapest compute you’ll own for the foreseeable future is whatever is already humming in your rack; treat it accordingly.

Stay Frosty,

Andy

SysAdmin Weekly

• The AI doom narrative has hit saturation. Some of the doom is warranted. A lot of it is marketing.

• “AI” is doing PR work for layoffs that were really overhiring corrections. The investor signal is the point.

• Episode 046 of the podcast is out: the honest case for Claude Code in SysAdmin workflows, plus three fresh Linux LPE CVEs.

• Tool of the Week: Ollama. Run open-weight models locally, no data leaves your network.

• A full data breakdown of the doom narrative is coming on SysAdmin Weekly Podcast Episode 047 with Eric Siron in a day or two.

From the Console

The AI doom narrative has hit saturation. You feel it on LinkedIn. You see it in every other headline. Microsoft’s AI chief telling lawyers, accountants, and marketers they have eighteen months. Big tech CEOs floating that AI will replace half of white-collar work by 2030. Investors clapping. The rest of us reading the room.

Some of the doom is real. The transition is going to be painful for real people. Data center water usage today is a problem. Real jobs have been cut and real households are absorbing that. I am not minimizing any of it.

The rest of the doom is marketing dressed up as inevitability. And it is loud enough that even people who already do this work for a living are starting to second-guess themselves. That is the part that bothers me.

So I went looking at the actual data. Layoff numbers. The WEF Future of Jobs Report. Energy and water projections. What the hyperscalers are actually doing about it versus what the headlines say they are doing. There is a full breakdown coming on the podcast with co-host Eric Siron in the next day or two. Until then, the short version: the data does not say what the headlines say it says.

The rest of this issue is about the part of the doom narrative that does not get enough scrutiny in our circles: how it is being used.

And now, back to our regularly scheduled programming.

The Latest on the SysAdmin Weekly Podcast

Episode: 046 - Can Claude Code Help SysAdmins? Scripting, Log Analysis, and the CLAUDE.md Workflow

Topic: Andy makes the honest case for Claude Code in SysAdmin workflows, with the caveats and the parts that still fall short. Plus three recent Linux kernel local-privilege escalation vulnerabilities and what they mean for your patch cycle.

Reasons to tune in:

• Most AI demos are built for developers. This is a long-time SysAdmin walking through what actually helps and what does not

• The CLAUDE.md workflow is the single biggest thing most folks are missing when they try AI in their day job

• Three Linux LPE vulnerabilities (Fragnesia, DirtyFrag, CopyFail) dropped in quick succession. Local-only, but still on the patch list

If you prefer to read about my experiences with Claude Code instead of the podcast episode, I did a write-up over on my blog at andyontech.com

All Links For the Show: https://www.sysadminweekly.com or watch / listen below!

The Take

The piece of the AI layoff story that does not get enough airtime is the part where AI is the excuse and not the reason. Again, not downplaying those jobs that were legit replaced by AI, but the enterprise space seems to love making AI the scapegoat, warranted or not.

The early wave of 2023 tech layoffs had a different reason on the slide deck. IBM and a few others actually said it: we over-hired during COVID by something close to thirty percent, the demand correction hit, and we cut. That is not a comfortable thing for an executive to say in public. It also is not an investor-friendly thing to say at an earnings call. So the language quietly changed.

By 2024 and into 2025, the same kinds of layoffs were getting a different label. “Restructuring around AI.” “Reallocating capital toward AI initiatives.” “Aligning our workforce with the AI-first future.” Conveniently, that wording does two things at once: it makes layoffs feel like strategy instead of correction, and it signals to investors that the company is an AI play, which moves the stock.

We are not just speculating here. Cisco cut roughly 4,000 people on the back of strong earnings. Oracle is reportedly cutting up to 30,000 jobs (about 18% of its headcount) while standing up a massive data center buildout. Meta announced 8,000 more cuts in May while telegraphing $115B to $135B in 2026 capital expenditure, almost all of it AI infrastructure. Read those press releases against each other and the pattern is consistent: trim the headcount line, route the savings into AI capex, take the investor bump for being “AI-first.”

Again, that doesn’t mean AI is NOT displacing some jobs. It is, in specific roles, at specific companies, on a real timeline. It does mean, though, that “we cut X people because of AI” is doing a lot more PR work in 2026 than it is doing operational work. And if you are a SysAdmin, an MSP operator, or an IT manager trying to read the temperature in your own organization, that distinction matters. The internal memo and the external press release are not telling the same story for the same reason.

The piece I keep coming back to is something Eric Siron mentioned in a recent recording of the podcast….. when an executive who has never been an accountant tells accountants they have eighteen months, ask that executive what THIER numbers actually need to look like by then. Eighteen months is often less about when the technology will be ready and more about when the latest AI bet has to start showing returns.

We will get into the data side, the WEF Future of Jobs numbers, the historical tech wave precedents, the water and power question, in detail when Episode 047 drops with Eric Siron. For now: if your CIO walks in citing the eighteen-month deadline, ask them to source it. Real conversations get easier from there.

Community Signal

High-signal community work worth your attention this week.

Brandon Lee (VMware vExpert) - “I Built a Local AI Coding Agent Home Lab Setup With OpenCode and Ollama” - The practical antidote to the doom narrative… practical use of AI systems! In this post Brandon Lee walks through standing up OpenCode against a local Ollama instance, with the model running on his own hardware and no data leaving the lab. The part worth the read is the honest scoping: where the local setup actually competes with the cloud offerings, where it does not yet, and what hardware you need to make it usable. Practitioner work, no vendor influence.

Tool of the Week

Ollama - Run open-weight LLMs locally. Single binary, simple model pull syntax, OpenAI-compatible API surface for any tooling that already speaks that protocol.

The pitch in 2026: if you are tired of either feeding sensitive data to a hosted model or being told you are not allowed to use AI at all, Ollama is the third option. Pull a model (Llama, Qwen, Mistral, DeepSeek, Codestral, others), point your client at the local endpoint, and run inference on your own hardware. Pairs cleanly with OpenCode, Continue, and any IDE plugin or CLI tool that supports a custom OpenAI-compatible base URL.

To be straight: the local model quality at consumer hardware tiers is not the frontier-model quality you get from Anthropic or OpenAI. For privacy-sensitive workflows, internal-only use cases, log triage, and scripting against data you cannot ship outside the network, the gap closes considerably. For greenfield engineering work, you will still reach for the hosted option more often than not based on my experience. Either way, nice to have the options.

Quick Win of the Week

Pick one “AI is replacing X” headline you have seen this month. Trace it back to the primary source. Is it a CEO quote, a press release, a paper, or a survey? Whose survey? How were the numbers gathered? You will be surprised how often the headline number does not survive the trip back to where it came from. Sourcing your own claims is one of the most useful skills a SysAdmin can keep sharp in 2026.

Fun Retro SysAdmin Fact

The first “AI is coming for your job” panic in IT was not in 2023. It was in the late 1980s, during the expert systems boom, when symbolic-reasoning platforms running on Lisp machines were supposed to replace whole categories of decision-making work. By the early 1990s the entire commercial expert systems market had collapsed into what historians of computing now call the second AI Winter. The Lisp machines went to museums, the panic dissipated, and the next wave of IT growth (the consumer internet) created more jobs than the AI Winter ever displaced.

From My Other Corner of the Internet

AndyOnTech

Can Claude Code Help SysAdmins? Real Use Cases for IT Pros - The companion write-up to Episode 046. Covers the CLAUDE.md workflow, scope-based settings, and where to stay skeptical with sensitive data and production-adjacent prompts. If the episode landed, the post is the version you can hand to a teammate who prefers reading to listening.

Thing I’m Re-Thinking

The assumption: When a major tech CEO makes a specific timeline claim about AI displacement, I should weigh it as informed analysis from someone close to the technology.

Why it no longer holds up: After a few iterations, it REALLY seems like a lot of those claims are not predictions, they are deadlines. Eighteen months is a number that lines up with when AI infrastructure spend has to start producing returns. The person saying it has a financial reason for that number, and that reason is not “this is the date the technology will be ready.”

What I am testing instead: Treating those statements as investor communications, not as forecasts. If the same claim comes from someone who has actually done the job being replaced, who is working at the keyboard with the tools, and who has no AI-adjacent stock price to defend, that is the version I weigh.

Until Next Week

If the doom is loud enough that you are starting to repeat it without checking, that is the moment to go check.

Stay Frosty,

Andy

SysAdmin Weekly

• Was at a conference recently. Half the room was on their phones. That’s not a people problem.

• The culture of IT quietly starves curiosity. And we as SysAdmins let it happen.

• I stood up Forgejo in my lab out of curiosity. It ended up replacing every GitHub component of my workflow. Zero regrets.

• Lab work isn’t a hobby. It’s how you stay dangerous.

• This week: do something that doesn’t have a ticket number.

From the Console

I attended a recent tech event in Texas which turned out to be a good event, had decent content, and a strong sense of community. At one point, I’m sitting there looking around the room….observing people as I’m want to do at times to gauge interest in a particular topic. I noticed lots of phones out, laptops open, heads down and my initial reaction was frustration, because I’ve been the guy up on stage talking, many times. Then I caught myself.

These aren’t checked-out people. These are DEPLETED people. The queue never stops. The alerts never stop. And somewhere in all that noise the thing that made most of us good at this job (the curiosity), the “I wonder what happens if I press this button” instinct, just quietly gets starved out.

That’s the thing worth talking about this week.

Folks that are regulars of the SysAdmin Weekly Podcast have likely heard me talk about the importance of curiosity for SysAdmins. I’ve been in the industry long enough now to know that SysAdmins used to have more time to be curious. We used to have more time for training. In fact, we’ve done our jobs so well over the last decade or two that tech and “digital transformation” have become core to many businesses. So much so, that the many businesses simply cease to function when tech systems are offline or impacted in some way.

Pair that with the fact that many organizations still view IT as a cost center. I still run into accounting heads that see IT as a black hole for money, and I still see leadership teams not giving IT teams the resources or the time needed to get the job done while leaving room for self-improvement or self-care. SysAdmins are overworked, and understaffed / underfunded, and it shows.

That all said, I would urge all of you to maintain that curiosity. Tinker in the lab. Check out that new community tool. Spin up that new game server. Most of us got into this industry because we have a deep love for the technology. Sometimes experimenting with new tech for no reason other than curiosity is all that’s needed to keep the bad parts of the job from overwhelming you and in the process it helps you keep your skills current and growing.

And now, back to our regularly scheduled programming.

The latest on the SysAdmin Weekly Podcast

Episode: 045 - Why Is It Always DNS? How DNS Actually Works, Breaks, and Gets Exploited Topic: Andy and Eric pull apart the full DNS resolution chain, Active Directory DNS integration, DNS security threats, and two war stories that will make you go check your fourth domain controller.

Why this one matters: - Most SysAdmins work with DNS every day without actually understanding how it works. This is the episode that fixes that - If you’ve ever said “have you tried flushing the DNS cache?” and quietly hoped it worked, this one is for you - DNS is also how attackers move data and talk to compromised machines. Understanding the protocol means understanding the threat

Episode Links

All Links:

https://www.sysadminweekly.com

YouTube:

Spotify:

The Take

I stood up Forgejo in my homelab recently. Self-hosted git forge, two-class repo model, nginx reverse proxy, internal HTTPS with mkcert. Did I plan for this to become my full git workflow? No. It started as curiosity. It ended up replacing every GitHub component of my day-to-day. GitHub no longer has a seat at the table, unless a repo has to be public and in that case I use repo mirroring so I still only have to interact with Forgejo at the end of the day.

On top of those benefits the curiosity that drove me to test out Forgejo to begin with enabled me to learn things I wouldn’t have learned any other way.

That’s the whole point. The lab isn’t always about the output. It’s about keeping the instinct alive. The instinct to dig in, break something on purpose, and figure out why it works. That muscle atrophies when you stop using it… and the environment most of us work in is perfectly designed to let it atrophy. Tickets, queues, SLAs, “what did you close today.”

It’s become a sad reality that the people who think lab work is a waste of time are usually measuring the wrong thing. They want deliverables. The lab doesn’t produce deliverables. It produces practitioners who don’t panic when something breaks outside the documentation.

This isn’t a generational thing either. I’ve seen 20-year veterans lose the curiosity instinct just as fast as someone two years in. It’s a culture problem. The fix is the same regardless of time in the chair: build something this week that nobody asked you to build. I’ll be money that two things will happen:

1. You’ll learn something new

2. You’ll enjoy the time spent doing so

So that said, what personal tech project are you interested in that you’ve been putting off? Maybe today is the day to start?

Community Signal

High-signal community work worth your attention.

mrlokans.work - “State of Homelab 2026” - A personal, opinionated stack writeup from someone who documents their gaps honestly: no backups, single NVMe, running costs under $10/month. The part worth reading is the SOPS with age encryption section as an upgrade over Ansible Vault for version-controlled secret management. Good real-talk energy, no vendor influence, highly recommend!

Brandon Lee - “How I Built a Self-Healing Home Lab That Fixes Itself” - Brandon Lee documents building an n8n automation pipeline that monitors CI/CD via Discord, uses an AI agent to identify failures, and automatically re-triggers failed pipelines without human intervention. Proxmox, Prometheus, Grafana, Docker restart policies. The interesting angle here isn’t the specific stack; it’s the idea that the lab itself can model the same automation discipline you’d want in production. Also a good read for this week!

Tool of the Week

Beszel - Lightweight infrastructure monitoring with a hub-and-agent model that actually fits the way homelabs and small shops are built.

Lightweight single binary, hub-and-agent model. Monitors CPU, memory, disk, network, and Docker/Podman container stats. The web dashboard is clean, alerting is configured with sliders and toggles instead of a rule language, and it doesn’t require standing up a full Prometheus stack to get meaningful visibility.

To be straight with you: this is a homelab and small fleet tool. No SNMP, no custom metrics, no HA, SQLite-backed. It is not a Zabbix or Prometheus replacement for enterprise environments with SLA requirements. For everything else, it’s the current community default for a reason.

Quick Win of the Week

Pick one service you interact with every day and actually read how it works this week. Not a tutorial. Not a YouTube walkthrough. The documentation or the RFC. Twenty-five minutes. The goal isn’t to become an expert; it’s to make contact with the actual thing instead of just operating the surface of it.

Fun Retro SysAdmin Fact

Before DNS existed, every host on the ARPANET was tracked in a single HOSTS.TXT file maintained by one person at the Stanford Research Institute and distributed via FTP. By 1983, the growth of the network had made manual updates impossible, which is what prompted Paul Mockapetris to design DNS. The specs he wrote in RFC 1034 and RFC 1035 are still the foundation of every DNS lookup happening right now.

Worth Your Time

Forgejo - User Documentation - If The Take in this week’s edition landed and you want to actually spin up your own git forge, the docs are legitimately good. The user docs index includes a getting started section.

Uptime Kuma - If you spin up Beszel, pair it with this. Beszel watches internal resource metrics; Uptime Kuma watches external service availability (HTTP, TCP, DNS, SSL). They cover different failure modes and the community runs them together for that reason. Worth mentioning that it also has a large number of GitHub stars….. meaning this one has been around long enough to trust.

Thing I’m Re-Thinking

The assumption: Staying current means following the news cycle.

Why it no longer holds up: Reading about technology and understanding technology are not the same thing. You can follow every CVE, every product announcement, every LinkedIn hot take… and still be completely lost the first time you have to operate something unfamiliar under pressure.

What I’m testing instead: One new thing in the lab per month. Doesn’t have to be big. Doesn’t have to be relevant to the day job. Just has to be something I haven’t run before.

Until Next Week

The queue will be there Monday. Your curiosity might not be if you keep telling it to wait.

Stay Frosty,

Andy

SysAdmin Weekly

· Big Tech is consolidating compute, cloud, and client hardware at an unprecedented pace and open source is increasingly the only viable counterweight.

· The Steam Deck is the clearest mainstream proof that open source can compete on quality, flexibility, and user freedom, even in a consumer market dominated by locked ecosystems.

· Open source isn’t just idealism. It’s a practical, career-relevant strategy for SysAdmins managing costs, avoiding lock-in, and maintaining operational control.

· This week’s Community Signal highlights open source tools worth building into your stack right now.

· Tool of the Week: A self-hosted alternative that punches well above its weight class.

From the Console

The Locked Room Problem

I want you to think back to a time… maybe recent, maybe not, where a vendor changed something fundamental about a tool you depended on. Maybe it was a licensing model. Maybe it was a feature that got paywalled. Maybe it was an acquisition that turned your go-to platform into a product roadmap you no longer recognized.

The most recent example for me, was Firefox. And if you’ve been listening to the podcast or have read my lengthy blog post on the subject, you already know how I feel about that particular saga.

That thing I keep coming back to: that feeling of helplessness….of being locked in a room where someone else controls the door, isn’t unique to Mozilla. It’s, sadly becoming the default state of modern enterprise IT, for a number of reasons.

In issue #21, we talked about the Great Compute Divide. Big Tech is hoarding GPUs, cloud vendors are pitching you on renting your own PC BACK from them, and RAM prices have quadrupled because the market for on-prem compute is being systematically starved (whether that’s the intention or not remains to be seen). That said, the consolidation is real, it’s accelerating, and it has consequences.

So what’s the counterweight? Despite my example above being Mozilla / Firefox (which has started catering to market demands) I’d argue that the long-term counterweight is in fact open source. It’s also worth noting that I don’t mean that in the idealistic, “free software should be free” sense (though that argument has merit). I mean open source as a practical, professional survival strategy for SysAdmins.

And if you want the clearest mainstream proof that this strategy works, look no further than Valve’s Steam Deck.

Here’s a consumer device running a custom Arch Linux distribution called SteamOS that ships with full terminal access, supports arbitrary software installation, and lets you configure, break, and rebuild it however you see fit. Valve didn’t lock it down. They handed you the keys. And the result? A device with one of the most enthusiastic and capable user communities in consumer tech, running games and workloads that “shouldn’t” work because the platform respects user freedom.

That’s the open source promise made tangible…. And it’s exactly the philosophy SysAdmins need to be applying to their own stacks right now.

This is the part where you all come at me with torches and pitchforks… “But, Andy! What about support”? I would make the argument that the support question can be worked around while vendor lock-in, hardware scarcity, planned obsolesence, and adversarial behaviour towards customers are much more difficult to contend with. At least with the support question you can:

· Train up internally

· Partner with a third party

· Some Linux distros have their own support mechanisms

The other issues I laid out above have more difficult solutions for the most part.

Whatever your stance on this issue is, I would urge you, at the very least, to conduct an audit of all functions within your environment where you depend solely on a single vendor. Which of those vendors have you locked in, in some way? Those are the areas you need to at least have a fallback plan in mind in the event that things go south.

A SysAdmin Clinging to an Open Source life preserver

And now, back to our regularly scheduled programming.

🎙 Recently on the SysAdmin Weekly Podcast

SysAdmin Weekly - 039 - BitLocker, Key Escrow, and the Microsoft Trust Question

All the relevant links for this episode can be found below:

· SysAdmin Weekly Website

· SysAdmin Weekly Github Discussions Board

The Case for Open Source in 2026 - Beyond the Ideology

I’ve been on a practicality kick lately with my content. So, let’s get practical for a second. Building on the “From the Console” section in this week’s issue of the newsletter, here’s why open source deserves a strategic seat at the table for every SysAdmin today:

1. You Own the Exit

Proprietary software comes with a quiet hostage clause, which is sometimes spelled out (and sometimes not). When the vendor raises prices, discontinues a feature, gets acquired, or simply decides your use case no longer aligns with their roadmap, you are stuck….full stop. Open source breaks that clause. You can fork it, self-host it, freeze it at a version that works, or migrate on your terms.

This isn’t hypothetical. Looking at a few examples:

· BookStack - Open Source Wiki / Documentation App

· Gitea - Self Hosted Open Source Code Repo Software (Note: owned by a for-profit org)

· Forgejo - Self Hosted Open Source Code Repo Software

· Nextcloud - Self hosted cloud filestorage

· Restic - Awesome CLI-based backup application

These are all tools that we’ve mentioned in this newsletter before and they exist because the proprietary alternatives became untenable for a meaningful number of organizations. The community voted with their forks.

2. Auditability Is a Security Feature

Putting my security brain in place for a second, I would also make the claim that, in general, you cannot fully trust what you cannot inspect. There are some exceptions to that rule for certain software suites, sure, but I feel it’s true when speaking generalities. The fact is, that with proprietary tooling, you’re accepting a black box at the foundation of your infrastructure. With open source, the code is accessible and viewable. Security researchers can (and do) find and fix vulnerabilities more quickly because the surface is visible. Now, it’s worth noting that this isn’t always comfortable. Visibility cuts both ways, but at least it’s honest, open, and everyone can see things for what they are, warts and all.

3. Cost Control in a World of Subscription Creep

SaaS subscriptions compound quietly. Open source tools, even with self-hosting overhead, frequently deliver better TCO at scale, especially for non-glamorous infrastructure services: documentation platforms, backup tools, monitoring stacks, identity providers.

4. The Talent Pipeline Recognizes It

The next wave of SysAdmins grew up on GitHub, Linux, and Docker. Open source fluency is increasingly a hiring signal, and building an open-source-forward environment makes your org more attractive to the people you want to retain.

Core Fundamentals: Reading an Open Source Project Before You Commit

Not all open source is created equal. Before you adopt a tool and build workflow around it, here’s a quick due diligence checklist:

· Commit frequency: Is the project actively maintained? When was the last commit?

· Issue response time: Are maintainers engaged with bug reports? Or are issues piling up unanswered?

· License: MIT, Apache 2.0, and GPL all have different implications for commercial use and modification. Know which one you’re working with.

· Community size: A project with thousands of GitHub stars and an active Discord/forum is less likely to be abandoned than a solo maintainer’s side project.

· Commercial backing: Projects with a company behind them (HashiCorp, Grafana Labs, Canonical, Gitea) have longevity. That said, watch for the “open core” model that some of them offer. Open core is where the good stuff is still generally paywalled.

· Documentation quality: Poor docs are a maintenance burden. Great docs are a force multiplier.

Community Signal

High-signal community content worth your attention this week > open source edition.

awesome-selfhosted – The Community’s Master List If you only bookmark one link from this edition, make it this one! Awesome-selfhosted is a community-maintained GitHub repository cataloging hundreds of free, open source network services and web applications you can host on your own infrastructure. Organized by category, it includes stuff like: analytics and backup tools to document management, monitoring, and identity providers…etc..etc. It’s the definitive reference for SysAdmins evaluating self-hosted alternatives to SaaS platforms. Think of it as the antidote to vendor lock-in, one category at a time.

Gitea Official Migration Documentation Full disclosure: There have been some community concerns around Gitea now existing under a for-profit company (See further down re: Forgejo). That said, Gitea remains a solid git repo hosting option that still retains the open source label. This documentation is a great resource for getting started if you’ve never used Gitea before. The official documentation covers installation, systemd service configuration, access, etc. That migration angle is the key differentiator here. It’s not just “here’s how to install a thing.” It’s “here’s how to actually move the thing.” If you’ve been meaning to get your repos off a platform you don’t control, this is your starting point.

TechReviewer – Linux Hits 90% Windows Game Compatibility Milestone The Steam Deck and Proton keep coming up in this edition for good reason. They’re the clearest mainstream proof that open source can compete at scale without compromising on user experience. This piece puts hard numbers to that argument: over 5.6 million Steam Decks sold by mid-2025, 21,694 Deck Verified games, and nearly 90% of Windows titles now running on Linux via Proton. The story here really shows what happens when a platform holder commits to openness and invests in the community infrastructure to back it up. Sound familiar?

Tool of the Week

Forgejo - A community-driven fork of Gitea, born after concerns about Gitea’s governance direction under a new for-profit organization. Forgejo is a fully self-hosted Git service. Like Gitea, you can think of it like GitHub, but it’s yours and it lives in your environment. Lightweight, fast, and increasingly the preferred choice for SysAdmins who want source control without handing data to Microsoft.

Quick Win of the Week

Audit one proprietary tool in your current stack this week. Ask these three questions:

1. What does it cost annually (licensing + admin overhead)?

2. Is there a credible open source alternative?

3. What would migration realistically take?

You don’t have to move anything. Just know your options. Because the time to evaluate alternatives is before your vendor sends you a renewal notice with a 40% price increase.

Fun Retro SysAdmin Fact

The GNU Project, which is the foundation of modern open source as we know it, was announced by Richard Stallman in September 1983. That means the open source movement is older than most of the Windows Server versions you’ve probably had to decommission. And unlike those servers, it’s not going anywhere.

Until Next Week

The tools to stay independent already exist. They’re battle-tested, community-supported, and increasingly production-ready. The question isn’t whether open source is viable, the Steam Deck is one project proved that argument is over, in my opinion. The question is whether you’re building your stack in a way that keeps the exit door unlocked.

Thanks for reading and stay frosty out there!

Andy @

• Big Tech is consolidating massive amounts of compute while AI workloads continue driving unprecedented demand for GPUs and memory.

• Major cloud vendors are openly advocating for a future where you rent your PC and infrastructure instead of owning it.

• RAM prices have exploded, with some configurations quadrupling in cost, effectively pricing out homelabs and even certain enterprise deployments.

• The more workloads move to centralized cloud platforms, the less independent control organizations retain over their own compute.

• This week’s From the Console asks a hard question: does compute consolidation ultimately reduce flexibility, choice, and long-term autonomy?

• Community signal worth your time: a practical guide to estimating and managing Azure costs before they spiral out of control.

From the Console

I think it finally happened….. I’ve finally circled back to the portion of the industry that prefers to run (most) workloads locally as opposed to “cloud-first”.

Trust me… this comes as much as a surprise to myself as it does you (or maybe not?). It was a very specific line of thinking and industry events that led me back down this path… events which we plan to cover in a future podcast episode. That said, I thought articulating them here would make for an interesting “From the Console” opener for this week’s Newsletter.

So what led me down this path? A number of different news stories over the past couple of months eventually led to a thought experiment… which, in turn, led to my new stance, I suppose.

So, what were those news stories, you ask?

Big Tech Helping Trump & GOP in the 2026 Midterm Elections

https://www.washingtonpost.com/technology/2025/12/19/tech-super-pacs-midterms-ai/

Now, first off, this newsletter is NOT a political newsletter. That said, we have mentioned politics on the podcast from time to time as it relates to the technical topics of the day. Regardless of your stance on the current state of US politics, the important and relevant bit for this discussion, is this personal take I put together re: the article:

There are a group of individuals with direct or indirect ties to “Big Tech” that are using their resources to influence a specific result(s) in the US 2026 Midterm election. When the same entities that control massive compute resources also shape political outcomes, it raises questions about long-term digital independence.

Political activism by big tech isn’t a new thing. It’s been going on for years now, but it’s an important item to point out as currently happening in our industry and it’s important to take into account for this editions “From the Console” discussion.

Quote from Jeff Bezos on Local Compute

https://www.windowscentral.com/artificial-intelligence/jeff-bezos-says-the-quiet-part-out-loud-bezos-envisions-that-youll-give-up-your-pc-for-an-ai-cloud-version

I came across this article from Windows Central in my usual RSS meanderings and Bezos’s stance on local compute is what caught my eye. This section of the article specifically.

So, what prediction did Bezos make back then, that seems particularly poignant right now? Bezos thinks that local PC hardware is antiquated, and that the future will revolve around cloud computing scenarios, where you rent your compute from companies like Amazon Web Services or Microsoft Azure.

There has been talk for some time now from the major cloud vendors talking about the “benefits” to “renting” a PC in the cloud. In fact Microsoft has even produced a line of endpoints (which are basically thin-clients) called Windows 365 Link who’s sole purpose is to connect to a cloud-hosted PC in Azure.

We’ve seen the massive consolidation of server-related workloads being pushed into large cloud environments for the last 10 to 15 years. Now, it seems, there is a renewed push to get a percentage of client market share hosted in large cloud providers as well.

Well Known and Ongoing Industry Memory Scarcity

https://tech.yahoo.com/computing/articles/ram-prices-exploding-heres-why-111115489.html

It’s become well known throughout our industry (and amongst the gaming community as well) that if you need to buy memory, do it NOW. Prices have been rising for some time and I can speak from personal experience on this one. About a year ago, I had built a decent box for some homelab workloads that has 96GB of memory in it. Out of curiosity, I checked what that same 96GB of memory would cost me today and I discovered that the cost for that given configuration has QUADRUPLED in price. I’ve even heard stories from fellow enterprise admins that their organizations have been effectively priced out of certain configurations due to the massive cost increase of memory.

We’ve seen the industry’s push to leverage cloud compute more and more. That said, there has always been a cohort of SysAdmins that have either been unable, or simply do NOT want the loss of control that using someone else’s hardware entails.

What Concerns Does This Raise?

Having consumed all three of these news stories in close succession, my brain started something of a thought experiment with some key points:

· The current industry (and arguably geopolitical) line of thought is: AI = Inferencing = Power

· We know that AI workloads require a gargantuan amount of compute resources

· There has been increasing market inertia to move workloads to the cloud

· Increased cloud dependence REMOVES the need for on-prem / owned compute hardware

This pattern ultimately takes us to one place, and that is a world where the hardware needed for massive compute workloads (like large-scale AI) is largely in the hands of a few major players.

The question I leave you with today is this:

Does the consolidation of compute, and by extension, the ability to run compute-heavy workloads (like AI), eventually take away choice, flexibility, and independent operation absent influence from big-tech & politics? And, does this sequence of events take away power from individuals by extension?

Also worth asking:

What happens when access to high-performance compute becomes a subscription privilege instead of an owned capability?

We’ll be sure to cover this in a future episode of the podcast with more discussion. However, for now, I’d LOVE to hear what your thoughts are on this! Feel free to comment below!

Big Tech Hording All the Compute

And now, back to our regularly scheduled programming

🎙 Last Time on the SysAdmin Weekly Podcast

Episode: SysAdmin Weekly - 038 - Making Security Decisions Based on Data, Not Fear

SysAdmin Weekly - 038 - Thumbnail Image

Topic: Given the state of tech marketing and the Cybersecurity news-cycle currently in the industry, this topic is increasingly important for today’s SysAdmins. It’s easy to fall into the trap of FUD (Fear, Uncertainty, Doubt) when it comes to Cybersecurity. The reasoning is rather simple. While we SysAdmins know that security is a necessary component of any modern network, the psychology behind the NEED for security comes from a place of danger and uncertainty. We must protect our networks against threats both real and suspect, so when a vendor comes knocking, saying “What if?”, it’s easy to to fall down that rabbit-hole.

In this episode Paul and Andy discuss steps you can take to make sure that you’re making these choices from a place of data and setting the fear and emotion to the side. Links to the applicable platforms can be found below:

· SysAdmin Weekly Website

· SysAdmin Weekly - 038 - on YouTube

· SysAdmin Weekly - 038 - on Spotify

· SysAdmin Weekly - 038 - on Apple Podcasts

Also, if you’ve got thoughts to share on this episode be sure to join the SysAdmin Weekly GitHub Discussion Boards!

Community Signal

I’ve got one Community Signal item for readers this week, and it’s high quality.

Thomas Maurer - Azure Cost Estimation

Beings compute consolidation and big public cloud platforms were a big discussion earlier in today’s edition, this resources felt applicable. I know MANY SysAdmins that are dealing with cost management on public cloud platforms and it can be really difficult to get an accurate estimation of your cloud-spend, especially when rolling out new services.

Thomas Maurer always provides fantastic resources and this post is no exception. I highly suggest you check it out!

A Backup Tool Worth Your Time

Restic is an open-source, cross-platform backup tool designed for simplicity, security, and efficiency. It performs fast, incremental backups using content-defined chunking and deduplication, meaning only new or changed data is stored after the first run. All backups are encrypted by default using strong cryptography before leaving your system, so even the storage provider cannot read your data.

Restic supports a wide range of backends including local disks, SFTP, SMB, and cloud object storage such as S3-compatible services, Azure Blob, and Backblaze B2. It is lightweight, script-friendly, and reliable, meaning it’s AWESOME for SysAdmins and homelab enthusiasts.

Official documentation is available HERE

From AndyOnTech

Why You Should Consider Debian Linux in 2025

I’ve spoke about this particular post in the newsletter in the past, but I’ve seen a number of “Why Debian” related posts on Reddit as of late, and even ran into the question on social media in the last week or so as well.

I wrote this article last year, so it’s a bit dated, but all the reasons why you might want to consider Debian Linux all stand in 2026. If you want the gritty details (and a YouTube video!), you can check out the link. Otherwise the short list is here:

· Stability

· Customizability

· Free and Open Source

· Highly Secure

· Fantastic Community

· Excellent Server Performance

· Wide Hardware compatibility

· Long-Term Servicing Options

If you find yourself currently hopping around and testing distros, I highly suggest you consider Debian. New features to do lag from time to time, but Debian can’t be beat in terms of rock-solid stability!

Until Next Week

I want to make sure I say it again. We here at SysAdmin Weekly REALLY appreciate all of you that partake in our content and community. As this weekend brings us the Superbowl in the US, I wish you a quiet weekend and may all your systems stay running, at least until the big game is over.

Until next week, stay frosty!

— Andy @

• Incident response plans and backup / DR strategies are boring, neglected, and absolutely career-saving when things go sideways. Review them before you need them.

• New podcast episode out: When Incident Response Plans Meet Reality, covering what actually belongs in an IR plan, who should be involved, and how tabletop exercises should work in practice.

• Curiosity isn’t optional for SysAdmins. Using short, intentional learning sessions can help you keep skills sharp even when motivation is low.

• Community signal worth your time: strong arguments for using Markdown and the CLI to improve focus, consistency, and long-term maintainability of technical documentation.

From the Console

With it still being somewhat near the start of the year, we’ve been reminding our listeners that it’s a good time to take a look at those…. foundational SysAdmin things that we tend to neglect. This would be things like Backup & DR planning, incident response plans…. recovery testing…etc.

Now to preface this opening section of this week’s newsletter I’m not judging anyone here. I get it. End-users are pounding on your door, mail services went offline AGAIN, and you’ve got a project list as long as the help output for ffmpeg. The last thing you want to think about are incident response plans that should still be good.

Also relevant along these lines and interesting, is our watch / listen metrics for last week’s episode aligns with this mindset as well. In our episode last week Eric and I discussed Incident Response Plans in the Real World and the consumption of that episode was noticeably less than normal. Again, I’m not knocking anyone, it’s just an interesting data point that supports my theory that the average SysAdmin either doesn’t have time for incident response / backup & DR planning or its boring and / or overdone in terms of content, or everyone assumes existing plans are…. fine.

I have to say, that in terms of interest and “general IT priorities”, incident response planning isn’t the sexiest topic, but it is a critical and foundational one. As SysAdmins, it’s our job to insure systems uptime and data safety whether that (thankless) work is visible to the organization or not. An (at minimum) annual review of your incident response and backup & DR plans does a couple things:

· Highlights technical gaps or limitations within critical business infrastructure

· Informs key stakeholders within the business that may have moved / changed / forgot about these initiatives

· Helps prevent overly long outages

· Safeguards company data

· Makes you look like the superhero and INSTANTLY validates your salary the second something happens and you’ve got the magic plan that saves everything

What’s the saying?

“An ounce of prevention is worth a pound of cure.”

Block some time off in the calendar, and soon. Again, I know these items aren’t interesting. They aren’t some shiny new tech that needs to be installed. They aren’t business transforming things, but you’ll be much better served to have them ready (and updated) and not need them, than the other way around. Chaos without a plan is just chaos. Chaos with a plan is still chaos, but at least it’s managed chaos where you come out the hero, and your organization stays out of news.

If you’d like some real world guidance on the review of your incident response plan and backup / DR plan, be sure to check out last week’s episode of the SysAdmin Weekly Podcast listed below!

Also, if you’d like to chime in with your incident response / backup & DR success (or horror) stories, check out this thread on the SysAdmin Weekly GitHub Discussions Board!

A SysAdmin ignoring the incident response plan in favor of cool infrastructure projects

And now…. back to our regularly scheduled programming.

🎙 The latest on the SysAdmin Weekly Podcast

SysAdmin Weekly - 037 - When Incident Response Plans Meet Reality

SysAdmin Weekly - 037 - When Incident Response Plans Meet Reality

Topic: In this week’s episode Andy and Eric discuss one of the most groan-inducing and seemingly boring SysAdmin Topics on the planet: Incident response plans. While many SysAdmins dislike the documentational nature of building (and maintaining) incident response plans, they are one of the most critical components to ongoing operations. To put it shortly, when you need it, you NEED it.

This episode gives you the details on:

· What to include in your incident response plan

· What stakeholders within your organization to involve

· Frequency and details covered during table-top exercises

· Lots more!

The episode also includes the usual nerd-hour talk, VMware / Broadcom gripes…etc.

Episode Links Here!

· All platform links

· SysAdmin Weekly - 037 - On YouTube

· SysAdmin Weekly - 037 - On Spotify

· SysAdmin Weekly - 037 - On Apple Podcasts

The Take - Curiosity Should be a Weekly Mandate for SysAdmins

I wanted to make a short take this week to continue some commentary on the importance of curiosity as a core trait(s) needed by SysAdmins.

It’s easy in this industry to rest on your existing knowledge, but you’ll find that you fall behind pretty quickly. It can be difficult to focus on learning new skills on the regular, especially when it’s (maybe) a technical skill that you’re really not interested in, but is required due to things you have to support during your day job. I find that curiosity helps with the resistance in this situation.

There are certain technical skills that I’ve committed to training myself on over the course of 2026, but they aren’t always the most interesting (I’ll plan a future blog post on this). What I find works well for me is to spend a little time digging into something that I AM highly interested in prior to tackling the “un-interesting” skill. For example, I’ve long been curious about Unity. While I’m not officially a programmer, I dabble. On top of that, I’ve been playing video games now for nearly 40 years, which is basically how long I’ve been able to properly use a controller. As such, I’ve always had an interest in tinkering with game engines and maybe someday building my own game.

With that said, the sequence I’ve found that works well for learning those “un-interesting” skills is the following:

· Define a training time period

· Start with a short session tinkering / training on something that actively interests you. (Unity in my case)

· Actively work on the training / tech that you’re obligated to work on

· Repeat sequence as needed

Is this context switching? Yes… to a degree, and yes, I do hate context switching. But, session lengths should be long enough to where it doesn’t really have an impact. For example, I follow the pomodoro method for just about everything. In this context, I’ll tinker with unity for 25 minutes (one pomodoro) and then commit 2 pomodoros to the other training need.

This method helps make sure that I learn the skills that I need to learn for external reasons, while also keeping my brain engaged and in “learning mode”.

Hopefully this will be as helpful to you as it is to me!

Community Signal

High-signal community work worth your attention.

Ryan Elston – Why I Use Markdown and Why You Should Too

As I’ll shortly be singing the praises of Markdown later in this newsletter, this article by Ryan Elston seemed to be highly relevant. In this post Ryan covers why Markdown is a highly valuable documentational language and even provides a great argument for why you SHOULD be using markdown along with some markdown authoring tools to get started!

Mike Robbins - Write clear step-by-step Instructions Using Markdown

Mike Robbins is also a fantastic source of information about markdown as well as how to properly use it in a how-to format. It’s worth noting that Mike is also a great source of PowerShell scripting knowledge as well! His blog is highly recommended!

Question I Got Asked (and the Real Answer): “Why are you obsessed with the CLI and markdown?”

Boy, it’s been a very commentary heavy week for me throughout this edition, but I feel like this will all provide some value (at least I hope it does).

Those that watch / listen to The SysAdmin Weekly Podcast know that I am an absolute CLI and Markdown nerd. My co-hosts have even made fun of me during certain episodes, which is warranted if I’m honest about it.

It so happens that I have had this question levied at me several times over the past couple of weeks:

*“Why are you obsessed with the CLI and markdown?”

First, valid question. Second, I’ll plan on a long form blog post or YouTube video (maybe both?) that explains my method and the “Why?” behind it.

In short I author ALL of my content using the following tools:

· Kitty (A terminal emulator)

· vim (A CLI-based text editor)

· Git, GitHub, & Gitea (to act as change tracking methods & content repositories)

· Markdown (my documentational language of choice)

· Pandoc (for converting markdown content to finalized formats)

To keep it short and straight to the point here, the core reasons “Why?” are as follows:

· I’m an easily distracted person, and CLI UIs are clean, focused, and purpose-built.

· Markdown is a defined and rule-driven documentational language that should output in concise formatting regardless of the author, assuming markdown principals are followed.

· As part of my job and my tech community efforts I spend A LOT of time authoring content of various types.

· A CLI environment paired with markdown prevents context switching, enables focus, and insures unified and consistent output.

· By using Git and Git-compatible repos I get the full benefit of code-review, pull requests, and commit history through the lifespan of the content

I’ll plan on covering this in a future blog post in more depth or via a podcast episode, but if you have questions you can reach out to me on LinkedIn or feel free to start a thread on the SysAdmin Weekly GitHub Discussion Board!

Until Next Week

Thanks for hanging out with us for another week! May you remain curious in your SysAdmin journey, and may your packets continue to flow as planned throughout the weekend!

Stay frosty!

Andy @

⏩ TL;DR – This Week in SysAdmin Land

• Abstraction is great… until the dashboard lies to you. A real-world Kubernetes + Longhorn storage lesson on why “schedulable” doesn’t always mean usable.

• Running stateful workloads on Kubernetes still requires understanding the underlying storage mechanics, replicas, disk sizing, and node-level constraints matter.

• SysAdmin Weekly now has an official GitHub Discussions home for announcements, technical discussions, polls, and general sysadmin banter.

• New podcast episode out: The Hidden Dangers of Abstraction in Modern IT that covers why deep technical understanding still matters in 2026.

• Community signal worth your time: a grounded take on why you probably don’t need 10 Gbps home fiber, and a smart rethink of how GitHub Copilot should actually be used.

From the Console

I was reminded to embrace curiosity in IT again this past weekend

In perfect timing with our episode regarding abstraction in modern IT (more on that below), I was reminded to embrace curiosity and not blindly trust / follow abstraction layers in modern software.

As many who watch the podcast know, I’ve been slowly converting many of the workloads in my homelab into containers running on a (now) 4-node K3S (Kubernetes Lite) cluster. This project was brought on by a number of things:

· I’ve always leveraged platforms like Azure Kubernetes Service and wanted to get up to speed on running local Kubernetes

· I’ve got aging hardware in my homelab and getting everything to follow Hyper-V best practices is…. difficult

· Distributed Architecture has always been highly interesting to me

· Many of the workloads I run benefit from being stateless

· I’ve rediscovered my love of all things open-source recently, and have taken steps to start using more stuff from the FOSS ecosystem

While, yes, many of the workloads I run can benefit from being stateless, I have several workloads that MUST be persistent, and as such, require persistent storage for data, state, etc…etc. To facilitate this in a clustered and highly-available fashion, I’ve deployed Longhorn to the cluster to provide distributed block storage across the cluster.

It’s an amazing system really. Unlike hypervisor clusters that require high degrees of compatibility checks, fabric deployment & management, etc, Longhorn makes storage configuration STUPID easy. Once setup and configured on the cluster, anytime a new node is added to the existing cluster, Longhorn will automagically ingest the available storage. Storage is spread in a redundant fashion across all the available nodes and fault domains are automatically defined and configured. Abstraction at it’s finest. This is where some of my assumptions caught me by surprise. Here is my high-level storage status for the cluster:

Longhorn Storage Dashboard on K3S

At first glance you could make the (reasonable) assumption that:

· 459Gi Schedulable = Storage I Can Use

· 226Gi Reserved = Reserved Storage for Data Parity and Safety

· 68.3Gi Used = Storage that the cluster is actively using

Imagine my surprise then when I carved off 100Gi for a persistent Debian Apt-Cacher container that the volume would fail to mount or do anything meaningful. See, on the surface, everything seems to work. It seems like I would have PLENTY of room for this workload. However, this is where abstraction was lying to me.

To make a long story short, my limitation is NOT the 459Gi number. The REAL limitation is the smallest available schedulable disk on any node that must hold a replica after accounting for:

· Replica count

· Reserved space

· Disk pressure thresholds

This is information that which had not been presented to me on the dashboard. A 100Gi volume with three replicas needs ~300Gi total and each replica MUST fit entirely on a single disk. If one node can’t fit said 100Gi replica…. game over. No volume attachment for you.

Once I put my assumptions aside, got curious, and acknowledged that this was a case where an abstraction layer was hiding the truth from me, I got the manual out and really dug into how Longhorn handles it’s underlying storage. Once I had that deep technical understanding of the underlying storage system, I was able to make informed decisions without solely relying on the abstraction layer, and ultimately get my workload up and off the ground after making different architectural decisions.

It was sobering to run into the very dangers of abstraction that we had JUST gotten done talking about on SysAdmin Weekly so quickly after posting that episode (last week - see below). Also, to be fair to Longhorn, that information is surfaced within the dashboard, just not in THAT particular view. This is, again, how abstraction (and assumptions) can get you into trouble.

So remember, when you run into a situation where things seem one way, but you have that gut feeling that not all appears as it should, ask yourself if you have all the information. Do you truly understand everything under the hood? You’ll be a better SysAdmin for it.

And now…. on to our regularly scheduled programming!

SysAdmin Weekly Announcements

As I’ve mentioned in several episodes of the podcast, we’ve been working on a defined community “hangout” spot that we can use for announcements, community conversations, knowledge sharing, questions, polls…etc…etc. Well I’m happy to announce that said initiative is complete and we now have an official GitHub Discussions repository setup specifically for this purpose.

You can find the SysAdmin Weekly GitHub Discussions Repository Here!

Why GitHub Discussions?

My one hard requirement for any community gathering place was low barrier of entry. Applications like Discord / Slack, while good, require users to download an app and / or setup an account. MOST SysAdmins will already have a GitHub account and GitHub is widely supported in both desktop web browsers as well as on mobile.

The notifications system is simple as well. For example:

· If you just want a quick shortcut to the SysAdmin Weekly discussions repo, just star it.

· If you want granular notifications based on activity, “watch” the repo instead.

What Type of Content Can We Expect There?

You can expect to find:

· Show / Newsletter Announcements

· Discussion Posts for the latest episode / edition of both the podcast AND the newsletter

· Polls about content you’d like to see more of on the show

· Technical Discussions

· Q & A

· Helpful scripts / tools like I posted here

· Friendly Banter

· Etc…

What Do I Need to Do to Get Started?

Outside of setting up a GitHub account in the off chance you don’t have one? Nothing! Feel free to check it out and Introduce yourself, if you’d like! (optional)

🎙 The Latest on the SysAdmin Weekly Podcast

SysAdmin Weekly - 036: The Hidden Dangers of Abstraction in Modern IT

SysAdmin Weekly Podcast Episode 036

Topic: In our first episode of 2026, Andy and Eric dive into the topic of rampant abstraction in modern IT. While yes, abstraction layers have solved many scaling issues over the last decade, there IS a hidden cost. Do SysAdmins still have that deep technical understanding of the entire stack? This episode explores that question in depth!

Why this one matters:

· This episode highlights the dangers of relying heavily on abstracted features & deployments without deep understanding

· Should something break under abstraction layers, SysAdmins are often at the mercy of a vendor to solve the issue.

· This epsiode contains some tips and tricks to help you ensure that your skills are at a point where you can benefit from abstraction layers, but NOT be dependent on them.

Community Signal

Here are some super-cool community links I’ve stumbled upon recently. Highly recommended!

Didier Van Hoye – Do I really need 10 Gbps fiber to the home?

Didier cuts through the marketing noise around ultra-fast residential fiber and lands on a conclusion most practicing SysAdmins already know: raw bandwidth is rarely the real bottleneck. Using his own home setup as an example, he shows how a relatively modest fiber connection is more than sufficient for remote work, lab access, and day-to-day operations, especially when most heavy lifting stays local. The real value here is the reminder that stability, latency, and realistic workload analysis matter far more than chasing headline speeds, and that blindly upgrading capacity without a matching use case is just another form of abstraction-driven decision making.

Luise Freese – Stop using GitHub Copilot as a chatbot!

Luise takes a hard look at the common pattern of treating GitHub Copilot like a chat-based coding partner and explains why that workflow almost always feels fragile and slow. This is not because the model is dumb, but because the interaction model is unstable. Rather than prompt, re-prompt, and steer in chat until something sticks, the article argues that you get far better results by encoding stable constraints and task definitions in files before ever calling Copilot, so the model can actually execute against known invariants instead of chasing shifting context. This post reframes Copilot from a reactive autocomplete toy into a tool that thrives within explicit structure, which is a valuable shift in mindset for anyone trying to use AI in software development.

From AndyOnTech and Project Runspace

AndyOnTech

Regular listeners of the podcast will know I went on something of a…. “rant” during the last podcast episode of the year. In said rant, I explained how there are no longer any good web browsers available in the marketing today. I recently posted a lengthy blog post along the same lines if you’re interested. I’ve linked that below, along with the Podcast episode from late last year on the same topic:

· There Are No Good Web Browsers Left (and It Is a BIG Problem)

· SysAdmin Weekly - 035 - AI Browsers, Chromium Monoculture, and the Future of Browser Security

Handy Tool - Pomocli

Full disclosure: This week I am selfishly promoting my own tool. That said, I can hardly call it a “tool” and more of a “anti-context switcher”. Folks that know me know I’m a BIG markdown guy and love to create my content within the terminal. Context switching is highly impactful on my output, so I go to great lengths to prevent it if possible.

Awhile back I had released a very simple cli tool called Pomocli that acted as a terminal-based pomodoro timer. It would ask what you planned on working on, log that to a text file, then start a timer. I always liked it’s lightness…. but hated how it looked. I recently started tinkering with building basic UIs using Ncurses and Python. As a result, I decided to rebuild Pomocli with a curses-based UI that still fills the same role with the added feature of allowing you to browse previous work sprints.

Pomocli image

You can find Pomocli HERE

I hope you all find it as useful as I do!

Until Next Week

Thanks for reading this week’s edition! I really hope everyone is enjoying the new, more conversational format! If you have any feedback or would like to see a specific topic discussed, be sure to let us know on the SysAdmin Weekly GitHub Discussions Board!

May your systems stay running, and may you avoid maintenance at 5pm on a Friday!

Stay Frosty!

--Andy @ SysAdmin Weekly

· The newsletter is evolving! This means fewer link dumps, more opinion, commentary, and real-world SysAdmin pain from the trenches.

· Theme of the week: Craft vs. Shovelware and why modern software feels more brittle, less thoughtful, and far more hostile to operators.

· A real-world rant with receipts: Microsoft Authenticator quietly falling over once you manage a lot of accounts… and the only fix being “nuke from orbit and start over.”

· The latest SysAdmin Weekly Podcast digs deeper into “good enough” software and why SysAdmins are the ones paying the price for it.

· Community Signal highlights thoughtful takes on whether software is actually getting worse. Plus practical Azure Local update guidance you’ll actually need.

· Other links worth your time this week include one rock-solid security podcast and one non-technical sanity-preserver (because balance matters).

· Looking ahead: plans to spin up GitHub Discussions as a low-friction, SysAdmin-friendly place to keep conversations going beyond the podcast and newsletter.

An Update on the Newsletter

First off, I’m sensitive to the fact that we’ve been away on this newsletter for quite sometime (4 weeks?) and I’m also very aware that I’ve provided MANY “Updates on the Newsletter” like this over the last month or two. The fact of the matter comes down to two things:

1. With the day job, the podcast, and a VERY time intensive side-project I’ve been working on, getting to this newsletter has been inconsistent and difficult.

2. I haven’t been 100% feeling the “vibe” of the newsletter format we’ve been using up until this point. Statistically “Link-dump” newsletter are one of the most common and while, yes, they can provide some value depending on the quality of the links, dumping a collection of links each an every week just to provide a collection of links doesn’t produce CONSISTENT value that I personally find worthy of your time.

Moving forward this newsletter will serve as much more of a commentary and opinion-based newsletter remaining laser-focused on SysAdmins and the challenges they face in the trenches every day. Will there still be some links? Sure, but the format will change to content that is much more conversational in style which provides a better backdrop for me to produce that CONSISTENT value that I want to provide this community.

If you have comments or concerns about the new direction, do let me know in the discussion section below!

From the Console

Our theme for this week’s edition of the newsletter focuses on “Craft vs. Shovelware”. I’ve been in the industry long enough now to recognize that software seems to be more brittle and problematic now than it used to. The amount of times that I’ve had to employ some sort of workaround to a seemingly simple-to-solve bug in order to keep production live has certainly increased over the years. And yes, our environments are wildly more complex than they used to be what with cloud and hybrid deployments and yes, that introduces EXTRA chances for breakage. This does not however account for cases where applications operate in a way that is woefully unhelpful to the user.

I’ll provide my own example on this one. I have been a LONG time user of the Microsoft Authenticator app for MFA and Identity services across a number of tenants in Entra ID and various 3rd party accounts. As of this writing I have around 43 different accounts managed within the Microsoft authenticator app. One day, I tried to add account number 44 to the app and I get hit with a big nasty “device token error”. Long story short, it seems I’ve hit a community-experienced, but non-Microsoft documented issue with the authenticator app where either there is a some soft-limit on number of accounts or there is underlying token corruption. Either way, I am unable to add any more accounts until I rip and restore ALL 43 accounts currently in my authenticator app….. needless to say… I was NOT impressed.

As mentioned, Microsoft has never documented a maximum number of accounts supported by the Microsoft Authenticator app (that I’ve been able to find). However, there is substantial real-world community evidence that the app does not scale reliably for users managing dozens of tenants / accounts. Admins frequently report that once they reach a high number of accounts, Authenticator begins failing to add new accounts or register push notifications, often producing vague registration failures or “device token errors”. One commonly cited message states: “Error changing device token. You must delete and reactivate your accounts to correct the issue,” effectively forcing a full MFA re-registration across all accounts.

Microsoft’s public documentation focuses instead on per-user Entra ID limits (such as the five registered authenticator devices limit), while offering little guidance on app-level scalability or token exhaustion. Community discussions on Microsoft Q&A and Reddit consistently describe these token-related failures and the same recommended remediation of “delete and re-register” suggesting an undocumented Authenticator scalability or push token management issue rather than isolated user misconfiguration.

Which brings me to my point of this whole rant about software quality in the modern day and user-expected app behavior. I’m completely fine with imposed limitations on software (such as max number of supported accounts) as that is something that I, as a SysAdmin, can plan and architect for. What I would have expected in this situation, if my suspicions about scalability and token issues are correct, is a warning at ~30 accounts “Max number of accounts reached”. Instead we get silent breakage, and to quote Ultron from Avengers: Age of Ultron,…“Boom… the end… start again”.

All this is to say, software development needs to once again be given the time and consideration to be a “Craft” again. What we currently seem to have is a hastily built software “content-pipeline” that is designed to push features extra-fast and squeeze maximum revenue from every byte. Something’s got to give.

And now, back to our regularly scheduled programming.

🎙 The Latest on the SysAdmin Weekly Podcast

This week I had regular co-host Eric Siron back on the show to discuss the very issue of “Good Enough” software mentioned above. You’ll find that it’s deeply attuned to this week’s theme.

SysAdmin Weekly Episode 034 Thumbnail Image

Episode: “Good Enough” Software is Ruining IT (and SysAdmins are Paying the Price)

Topic: Software enshitification has been a real issue in the industry lately and there are real consequences (both spoken and not) of that fact. One of those consequences is the fact that SysAdmins are usually the ones that are facing the full brunt of poorly written software that was deemed “good enough” at some point. In this episode Eric and I (Andy) discuss the state of software in our industry while lamenting a time when software was a craft, and NOT just a content pipeline.

Episode links below:

· SysAdmin Weekly on YouTube

· SysAdmin Weekly on Spotify

· SysAdmin Weekly on Apple Podcasts

· SysAdmin Weekly Website

Community Signal

While researching the aforementioned issues regarding this week’s theme. I did happen upon some useful community posts along the same vein or similar:

• This older post on the Stackoverflow Blog by Issac Lyman that asks the question “Is software getting worse?” Issac has a very fair and reasoned look at the question, and even though I’m not a developer, I greatly enjoyed reading it. I highly suggest you check it out.

• I also wanted to include some practical SysAdmin help from the community this week, starting with How to Keep your Azure Local Instances Up to Date by Thomas Maurer! More SysAdmin’s are finding themselves in a situation where they’re starting to rollout Azure Local, so this knowledge is a MUST have.

Other Links Worth Your Time

As I’m a regular podcaster, I often get asked which podcasts do I regularly listen too. I’ll provide one of my favorite technical podcasts, and one more…. hobby related.

• My technical podcast recommendation is very security related, and I’ve mentioned it on SysAdmin Weekly several times. That is the Security Now Podcast featuring Steve Gibson. Steve does an AMAZING job of explaining cybersecurity topics and when you listen to an episode of this show, you’ll learn more about the given topic then you ever thought possible! Highly recommended!

• My NON-technical choice has to do with my long-standing gaming addiction. I’ve been playing video games since I was old enough to hold a controller and I love listening to Remember the Game. Adam (the host) does an amazing job of keeping things interesting and funny. If you’re a long time gamer like me, you won’t regret putting this one on your list.

Building the SysAdmin Weekly Community

A final closing note on my efforts to continue developing our growing community. It’s come to our attention that we need a place that is quicker when it comes to communications. Something that can be used for announcements as well ongoing discussion. While we could certainly comment on each episode posting as well as the newsletter comment sections here, the goal is to have a centralized location where most of the community conversation happens.

As it stands, I’m currently looking at using Github Discussions for this purpose. Most SysAdmins will already have a Github account, and it doesn’t require a new app to be installed, so the barrier to entry is low. Be on the lookout for this new hangout in the coming weeks.

Retro SysAdmin Fact of the Week

Don’t forget. For years, many enterprise systems treated a two-digit year as “good enough” and it could have taken down the world.

The Y2K problem wasn’t a mysterious bug or media hysteria; it was the logical outcome of early storage constraints and yes, short-term thinking. To be fair, saving two bytes per record mattered when memory and disk were measured in kilobytes. Meaning developers stored years as YY instead of YYYY. This “temporary workaround” quietly became the norm. When that calendar rolled to 2000, systems didn’t fail because they were old, they failed because, let’s face it, no one expected them to still be mission-critical 10, 20 or 30 years later. Y2K remains one of the most successful IT and SysAdmin remediation efforts in history.

See? The concept of “we’ll fix it later” has been around for a VERY long time.

Until Next Week

Thanks for reading this week’s edition! If you haven’t yet, be sure to hit that subscribe button. It’s free and you’ll get useful, maximum-uptime quality content in your mailbox every week to help you in the SysAdmin trenches. Also feel free to share with another SysAdmin. For real…. we want to help as many people out of unintended systems outages as possible.

Additionally you can always find all the links associated with this newsletter and the podcast at www.sysadminweekly.com.

Until next week! Stay Frosty!

-Andy
SysAdmin Weekly

· We’ve got a truly spooky Halloween-themed podcast this week (because who doesn’t like horror stories with servers?).

· Major critical patch alert: CVE‑2025‑59287 in Windows Server Update Services (WSUS) is actively exploited - if you run WSUS, patch now.

· We’re revisiting network fundamentals with the OSI model - yes, dust off your layer cake, we’re serving up layer 1 to 7 for your next troubleshooting session.

· Strong community content this week: Linux stress-testing, Azure Local overview, a data-driven look at Apple Silicon CPU core frequencies.

· Tool of the Week: a slick terminal app that could become your new favourite toy.

· Quick Win & Retro Fact - because even infrastructure nerds deserve a little snack.

· Wrap-up with a plug for next week’s podcast (you know you’ll listen).

An Update on the Publication

I greatly appreciate all the readers of this newsletter and you may have noticed an absence of the publication for the edition that was to come out two weeks ago. As folks that watch the podcast know, there has been A LOT of projects (both personal and professional) on my plate over the last month or two. This time of year is typically my busiest time of year and sadly something had to give. That said, I’m through my backlog, and looking forward to continue providing regular editions of this newsletter once again!

This Week’s Insight(s) from Andy

Theme for this edition: “Patch & Pivot: From Fundamental Layers to Active Threats”
In this issue, we’re straddling both ends of the SysAdmin spectrum: going back to the bedrock with the OSI model (don’t roll your eyes yet) and facing forward into the very real, very active world of exploited vulnerabilities (yes, the WSUS one). The juxtaposition is intentional: whether you’re explaining Layer 3 routing to a junior or scrambling to fix a critical RCE in your update infrastructure, the mindset of “understand deep; act fast” is what differentiates the average from the excellent.

When it comes to cybersecurity, and more importantly achieving real security outcomes, the mantra “understand deep; act fast” can feel intimidating for those pivoting into the security space. But here’s the truth: cybersecurity pros are often the ones in the room who understand their tech stack *most* deeply.

Why? Because to truly secure something, you have to understand it from the kernel up. Every layer, every dependency, every quirk. Yet, in a world obsessed with abstraction, we’ve drifted away from the underlying mechanics that keep our systems alive. That blind spot? It’s exactly where threat actors love to play.

So how does that tie into this week’s theme Patch and Pivot? Patching is easy. Anyone can apply an update and move on. But pivoting? That’s where the real challenge lies. SysAdmins are now standing at the crossroads of IT and security, and the ones who thrive will be those who dig deeper the ones who move beyond “apply patch” and into “understand why it matters.” Because real defense starts where surface-level understanding stops.

So if you’re a SysAdmin today, don’t wait to be handed a security title, start learning the internals your tools hide. You can’t protect what you don’t understand.

And now, back to our regularly scheduled programming…..

Latest on the SysAdmin Weekly Podcast

This week’s episode: Halloween “SysAdmin Horror Stories” - join Andy as he and guests spin real-world tales where things went terrifyingly wrong in infrastructure, cloud, and on-prem. Yes, there’s ghosting, haunted servers, and maybe even a spectre of unmanaged firmware creeping through the data centre.

What You Missed on Last Week’s Episode of SysAdmin Weekly

Last week’s episode: University IT training vs real-world readiness - Andy and guest Clay Tamam dove into whether formal IT degrees truly prepare sysadmins for the trenches, or if bootcamps, on-the-job hacks, and real mistakes carry more weight.

Sneak Peek of an Upcoming Episode

Next week: Andy and Eric talk about the “enshitification” of tech vendors and how vendor behavior has shifted from partner to parasite, what happens when the vendor plays you more than you play the tech, and what SysAdmins can do about it.

From AndyOnTech

How to stress-test Linux - A clean, practical walkthrough on putting your Linux boxes through controlled chaos. Covers CPU, memory, GPU, and I/O stress testing with just enough explanation to understand what’s actually happening under the hood. It’s the kind of test you’ll wish you ran *before* production went sideways. Perfect weekend lab project for anyone running a home server or validating hardware stability.

Core Fundamentals

The OSI model explained - It’s old-school, yes, but still gold. Understanding each of the seven layers gives you the lens to pinpoint where things go sideways: from physical cabling (Layer 1) to application behavior (Layer 7).

Helpful Community Content

• How to restore deleted Entra ID conditional access policies and named locations - From Jan Bakker: a lifesaver when someone nukes your conditional access by accident.

• Azure Local overview video - From Thomas Maurer: what Azure Local is, why you might need it, and how it fits with edge/distributed stuff.

• Updated CPU core frequencies for Apple Silicon Macs - Data-driven deep dive for the Apple side of the house. Cool geek reading.

Other SysAdmin Content from Vendors and Official Publications

· Fedora Linux 43 has been released - New features, new kernel, new life for test labs maybe.

· Windows 11 Insider Preview Build 26120.6982 (Beta Channel) announced - If you’re running lab machines or poking at future client OS behavior: heads up.

Security Headlines for SysAdmins

· Critical WSUS vulnerability: CVE-2025-59287 – patch immediately - This one is massive. A remote, unauthenticated RCE in WSUS, CVSS 9.8, active exploits. If you have a WSUS server role enabled: stop reading, go patch.

· Cybersecurity and Infrastructure Security Agency (CISA) updates their Known Exploited Vulnerabilities catalog - They’ve flagged the WSUS vuln among others; this isn’t hypothetical anymore.

Tool of the Week

Ghostty - A slick terminal-application you’ll want to try: for those nights when typing at 3 AM and wondering whether your shell can be cooler. Go ahead, indulge your inner penguin.

Quick Win of the Week

Check your inventory/CMDB and ask the question: “Which servers have the WSUS Server role enabled and are listening on ports 8530 or 8531?” If you find any, prioritize patching or isolating them immediately. You might just win this week before the next alert hits.

Fun Retro SysAdmin Fact

Back in the day, the architecture for the mainframe and minicomputer era relied on multi-user terminals connected via RS-232 lines (often at 9600 bps). That meant many sysadmins were night-shift heroes with endless string lights of terminal screens. Integrity of those lights vs. modern dust-filled racks? It’s evolution, baby.

☕ Wrap-Up

Thanks for riding shotgun with me through this week’s edition of SysAdmin Weekly remember: “Fundamentals + Vigilance = Survival.” If you only take one thing from this newsletter, make it patch early, understand deeply.

Don’t forget to tune into the podcast over at www.sysadminweekly.com and drop us a rating or share your own horror story (especially around midnight). Until next week: stay sharp, stay curious, and yes… stay spooky.

· Running on-prem LLMs gets real with Ollama

· Home lab talk and blueprints from Project Runspace

· PowerShell Select-Object deep dives

· Azure Hybrid Benefit for Linux explained

· A Study from OpenAI on LLM hallucinations

· The Jaguar ransomware story continues + sudo exploit headlines

· BookStack as a documentation MVP

This Week’s Insight(s) from Andy

This edition leans into a theme I’m calling “Reality Checks”. In this context that refers to the messy truth between our ideal architecture diagrams and what actually breaks in production. From AI hallucinations to credential leaks, active exploits, and big companies being breached, tech doesn’t always respect our boundaries. My note this week will tie those stories together and challenge you (and me) to stay humble, vigilant, and ready to adapt.

It’s funny that this edition also features podcast episodes that discuss lab build-outs and local LLM architecture discussions. It’s funny because I can think of several cases throughout my career where the architecture was initially made with assumptions but ultimately wasn’t the correct architecture in the end. One example that comes to mind is from when Microsoft RDS (Remote Desktop Services) was the new cool-kid on the block. The provider I was with at the time had been servicing a large customer who wanted to start leveraging the thin-client model and use co-located RDS services for end-users.

On-paper the architecture followed all Microsoft best practices and each RDS host was specced in a way that followed industry standards as well as those recommended by the customer’s business applications and developers. Well, it turned out that due to the way one of said applications consumed memory, it didn’t become clear what the full needed RDS host memory requirement was until switch to production. Even the software vendor claimed that all should be well on shared infrastructure like RDS. Spoiler alert…. it wasn’t. Even the small test sample prior to going live didn’t surface memory contention.

The result? Poor performance on launch day due to an unforeseen memory consumption issue. We had RDS servers crumbling under the weight of 3 to 4 times more memory usage per session than what the architecture called for. Needless to say, these systems were paging memory like crazy and performance was…. less than stellar.

The moral of the story? We can’t make assumptions and take vendor guidance at face value. Question everything and test everything. Challenge your own assumptions. Because at the end of the day, if you don’t, Murphey’s Law will challenge them for you…. and not usually in a way that you prefer.

And now… back to our regularly scheduled programming….

Latest on the SysAdmin Weekly Podcast

Running LLMs On-Prem with Ollama
Andy and Mike Nelson dig into what it looks like to host large language models in your own datacenter (or basement). They wrestle with resource demands, deployment strategies, use-cases, and integration headaches. If you’ve ever thought “AI → Cloud only,” this episode might shake your assumptions.

What You Missed on Last Week’s Episode of SysAdmin Weekly

Building a Home Lab on the Cheap
Andy and Eric walk through how to bootstrap a functional home lab without draining your bank account. From repurposed hardware to clever virtualization, it’s the kind of episode you revisit when your budget says “no upgrades this year.” Great lessons for both newcomers and seasoned admins alike.

Sneak Peek of an Upcoming Episode

SysAdmin Horror Stories (Coming Late October)
We’re stitching together a “hall of horrors” version of SysAdmin life, complete with outages, screw-ups, and lessons learned. Want your story immortalized? Send it to contact@sysadminweekly.com (shame optional). Be sure to anonymize as needed……just don’t leave the punchline out.

From Project Runspace

How to Build a Home Lab for Learning Server Administration
Eric’s post here is a companion to last week’s podcast: it breaks down physical vs. virtual choices, pitfalls to avoid, and how to structure a lab you’ll actually use. Think of it as your lab’s blueprint……don’t build blind.

Core Fundamentals

The Windows Boot Process
Here’s what actually happens when a Windows box powers on: first the Windows Boot Manager, then the OS loader, then resume loader (for hibernation/fast boot paths). All boot settings are tucked in the BCD, editable with bcdedit (or MSConfig for the less adventurous). Add in Secure Boot, Trusted Boot, and ELAM, and you’ve got multiple layers defending your system before the user even signs in.

Helpful Community Content

PowerShell Select-Object Deep Dive
Andrew Pla and Fred Weinmann dig into how Select-Object can be wielded for cleaner, more efficient pipelines. If your PowerShell scripts feel messy, this is the kind of polish you need.

Upgrade to Windows 11 25H2 via Intune
Joey Verlinden walks through using feature update policies in Intune to push 25H2. Its helpful if you manage many endpoints and want a smooth upgrade path.

Azure Hybrid Benefit for Linux Explained
Thomas Maurer and Shreya Baheti detail how you can bring existing Red Hat/SUSE subscriptions into Azure, cutting redundant licensing costs and potentially saving a boatload vs pay-as-you-go.

Other SysAdmin Content from Vendors and Official Publications

Why Language Models Hallucinate
OpenAI turns the microscope inward and explains how hallucinations arise from how models are trained and evaluated. The takeaway? Some hallucinations are baked into the design, not just sloppy inputs. Understanding this helps you build more resilient systems around AI outputs.

Security Headlines for SysAdmins

Jaguar / Land Rover Ransomware & Data Theft
Even massive firms get hit. Factories went offline, data was stolen, and initial denials gave way to a public breach. Lesson: size ≠ immunity.

Sudo Vulnerability Under Active Exploitation
The --chroot (-R) handling in sudo has a flaw being exploited in the wild. Local privilege escalation, bypassing sudoer rules — patch now, test quickly, and verify mitigation.

Tool of the Week

BookStack
A lean, self-hosted documentation platform built on Laravel. Supports WYSIWYG and Markdown editors, roles/permissions, and structured books/chapters. If you need a central place for runbooks or team docs, BookStack is a strong option without the overhead of massive wiki stacks.

Quick Win of the Week

Fire up a BookStack instance (Docker, VM, whatever), import one critical doc (onboarding, firewall rules, etc.), and force yourself to use it exclusively for one week. That tiny shift centralizes knowledge, surfaces gaps, and starts cracking down on siloed knowledge.

Fun Retro SysAdmin Fact

Back in the day, Microsoft’s BootVis was a tool for profiling and optimizing Windows boot performance. Now, we take near-instant VM boot and container snapshots for granted, but sysadmins still can’t resist chasing those milliseconds.

☕ Wrap-Up

That’s all for now. We’ve walked through AI realities, lab ambitions, boot internals, and real-world security failures. The disconnect between expectation and execution is where we earn our stripes. Stay grounded, stay skeptical, and never trust assumptions.

Don’t forget:
- Send your horror stories for October to contact@sysadminweekly.com - Share this newsletter with a SysAdmin peer
- Hit reply with your feedback, war stories, or what you’d like us to cover next

Stay curious,

Andy

· SysAdmin Weekly is moving to bi-weekly for deeper insights and less noise.

· New podcast episode digs into why documentation matters and how it saves your future self.

· Last week’s podcast broke down Conditional Access, including pitfalls, best practices, and lessons learned.

· Eric Siron shares tips on writing trouble tickets that actually get resolved.

· This week’s fundamentals spotlight is Linux basics, because cloud = Linux.

· Community voices highlight AI storage internals, running local AI agents, and Apple’s new disk image format.

· Apple preps macOS Tahoe with new fixes and updates for admins to tackle.

· Security headlines: worm-driven NPM supply chain attack and the Windows 10 support countdown.

· Tool of the week: Greenshot, the lightweight screenshot sidekick every sysadmin needs.

· Quick win: document one “failure mode” in your environment to save yourself later.

Schedule Change Announcement

SysAdmin Weekly - the Newsletter is going bi-weekly. Why? More value, less filler. This newsletter was always intended to be a supplement to the weekly podcast episodes and by doubling the time between issues, I can deliver insights and context instead of dumping a pile of links in your inbox. Quality over quantity…because your time is worth more than another unread newsletter.

NOTE: The podcast is our core content, and will remain (as always) on a weekly cadence!

This Week’s Insight(s) from Andy

This week ties together docs, tickets, lifecycle planning, and tooling. They’re not just chores they’re narratives. Documentation is the history book, tickets are the plot twists, OS lifecycles mark the chapters, and tools like Greenshot are the illustrations. Together, they tell the story of how systems live, break, and (hopefully) recover.

We’ve touched on learning in tech before, and I’ve found that storytelling, putting context around tech and solutions, is one of the most powerful ways to learn. Sure, the “cautionary tale” is classic IT lore, but there are also quieter stories worth telling: the “why do we do this” moments.

I think about the “why” a lot. Like when helping a nonprofit expand digital capabilities meant they could assist more people. Or when a medical customer’s patient database was down hard, and restoring it kept labs, imaging, and follow-up appointments running. Or in manufacturing, where IT-driven cost savings during a downturn literally kept entire teams employed.

Stories in IT aren’t just learning tools. They’re reminders of the very real human impact behind the job. So, the next time you’re buried in day-to-day sysadmin chaos, remember your “why.” Those stories you’ve lived? They matter.

And now… back to our regularly scheduled programming.

Latest on the SysAdmin Weekly Podcast

SysAdmin Weekly - 022 - Why IT Documentation Matters (Even if you Hate Writing it): Fresh episode just dropped today: Why Documentation Matters in IT. Eric and I talk about how documentation is your future self’s life raft, and that’s whether it’s recovery steps, config notes, or “who owns this service.” We share horror stories of missing docs causing chaos and give tips for making documentation suck less: structure, ownership, and findability.

What you Missed on Last Week’s Episode of SysAdmin Weekly

SysAdmin Weekly - 021 - Microsoft Entra ID Conditional Access Explained: Last week, Paul Schnackenburg joined me to dive into Conditional Access. What it is, why it matters, and where admins trip over implementation. We talked pitfalls (breaking workflows with too-tight policies), benefits (keeping accounts secure), and shared some real-world war stories. If you’re rolling out Conditional Access, this one’s your crash course.

Sneak Peek of the Next Episode

Coming up next week on SysAdmin Weekly: Building a Home Lab on the Cheap. We’ll be covering how to get started without draining your wallet. This includes hardware hacks, cloud freebies, and creative virtualization tricks.

I’d love to hear your homelab setups and stories. Share them with me, and I might feature a few in the show!

From Project Runspace

How to Submit an Effective Trouble Ticket - Eric shares some clear, actionable guidance on writing tickets that don’t get bounced around. Provide context, logs, what you tried, and what you expected. Better tickets = faster fixes and fewer headaches.

Core Fundamentals

Introduction to Linux (Linux Foundation Free Course)

Linux runs the world. This includes things like cloud workloads, containers, IoT, even your kid’s smart fridge. If you’re in IT and not comfortable on Linux, now’s the time. This free course gets you hands-on with the basics and helps you level up where it counts most.

Helpful Community Content

How Azure Storage Powers AI Workloads - Pierre Roman and Vamshi Kommineni explain how Blob Storage handles AI at scale. Storage isn’t just dumb disks anymore; it’s AI fuel.

Docker MCP Toolkit: Local Agentic AI Made Easy - Thomas Thornton breaks down running local AI agents with Docker’s MCP toolkit. Simplifies experimentation and orchestration for sysadmins tinkering with AI.

Apple’s New ASIF Disk Images - Howard Oakley benchmarks and explains who should care about Apple’s new format. If you deploy Mac images or deal with storage optimization, worth a read.

Other SysAdmin Content from Vendors and Official Publications

MacOS Tahoe Release Notes - New tweaks, fixes, and security updates. Admins managing Mac fleets should scan this closely for compatibility issues. Expect some patching fun and maybe a surprise or two in the UI changes.

Windows 10 End of Support - The end is near. If you’re still running Win10, plan upgrades now. Test legacy apps, drivers, and licenses before your helpdesk tickets explode.

Security Headlines for SysAdmins

Self-Propagating Supply Chain Attack Hits 187 NPM Packages - A worm-style exploit in npm packages. Mitigation tips: lock dependencies, use checksums, scan your supply chain, and restrict build agent permissions. Don’t let your CI/CD pipeline turn into a trojan horse.

Tool of the Week

Greenshot is Lightweight, open-source, and a sysadmin’s secret weapon for docs and tickets.

· Grab screen regions, windows, or even scrolling pages.

· Annotate, highlight, blur, and obfuscate sensitive bits.

· Export directly to clipboard, files, or email.

Greenshot is faster than Windows’ built-in tools and makes documenting issues painless.

Quick Win of the Week

This week’s theme is Infrastructure as Storytelling. Quick win: Pick one critical service and document its “failure story.” Who gets notified? What breaks? How do you fix it? Run the drill once and you’ll thank yourself when the lights go out.

Fun Retro SysAdmin Fact

SysAdmins in the ’80s swapped patches and tools on dial-up BBSs. Picture sitting by a phone line, listening to screeches, and praying no one picked up the phone mid-download. Kids today will never know the pain of a 2-hour patch download interrupted by mom making a call.

☕ Wrap-Up

That’s it for this round. Thanks for reading and keeping the lights on in IT.

New podcast goes live weekly: SysAdmin Weekly on YouTube — subscribe, share, and drop a comment.

Catch all past issues, resources, and extras from the newsletter at newsletter.sysadminweekly.com.

Stay caffeinated, stay patched, and stay sane.

- Andy

· 📅 Newsletter schedule shifting to mid-day Wednesdays (because mornings = ticket hell).

· 🎙️ New podcast: Top Microsoft Skills every SysAdmin should have.

· 🛡️ Security watch: Cloudflare caught up in Salesloft Drift mess + Microsoft deep dive on Storm-0501.

· 📩 Microsoft limiting email from .onmicrosoft.com domains.

· 💳 SSA database leak reminder: consider credit freezes (especially for kids).

· 🔧 Tool of the week: Pi-Hole (because ads are malware with better PR).

· 📰 Great community content: decision-making for SysAdmins, OpenStack POCs, career pauses, and retention policy scripting.

📢 Schedule Change Announcement

Heads-up! We’re moving the newsletter drop to mid-day on Wednesdays. Why? Because most of you are knee-deep in tickets, outages, and weird printer issues on Tuesday mornings. Consider this your mid-week coffee break instead of a Tuesday-morning distraction.

💡 This Week’s Insight(s) from Andy

If you think about it, most of a sysadmin’s job is building and maintaining guardrails.

Sometimes they’re technical, like Pi-Hole blocking junk traffic, retention policies keeping lawyers happy, Conditional Access stopping “Bob in Finance” from logging in from Timbuktu. Other times, they’re life-related, like freezing credit after a breach so the bad guys can’t open a mortgage in your kid’s name.

This week’s news is a reminder that attackers are constantly hunting for the gaps. Our job isn’t to stop every car on the road, it’s to make sure they don’t plow through our lane. Guardrails may not be glamorous, but without them, the crash is inevitable.

As many readers of this publication and subscribers of the podcast know, I’ve spent a good chunk of my career either as the customer of MSPs, working for an MSP, or selling to MSPs as a vendor. MSPs thrive on process and iteration, and the same can be said of most internal IT departments.

I’ve seen what happens when guardrails are missing. Take one customer I supported many years ago: they didn’t have any M365 rules in place to alert on the creation of inbox forwarding rules. A malicious insider with high-level IP access set one up that quietly forwarded sensitive data to a competitor. Without guardrails, security was an afterthought and abuse was inevitable.

Guardrails aren’t just about security though. RAID, for example, is a guardrail for your data. Without it, you’re a single drive failure away from disaster. Whether it’s data resilience or security posture, the guardrails we build and maintain are critical to the safety of both our businesses and the digital world we support.

Take that responsibility seriously. Build guardrails. Maintain them. And keep the chaos at bay.

And now…. back to our regularly scheduled programming.

🎧 Latest on the SysAdmin Weekly Podcast

Episode 020: The Top Microsoft Skills Every SysAdmin Needs
Eric and I break down the must-have Microsoft skills for today’s SysAdmins. Topics range from Entra ID know-how to keeping your PowerShell chops sharp. We talk about why old “checkbox” skills aren’t enough anymore, and which modern Microsoft tools really separate the heroes from the merely competent.

⏪ What You Missed on Last Week’s Episode

Episode 019: What Counts as a Security Breach (and What Doesn’t)
Last week we dug into the messy gray area of “breach” vs. “incident.” Not every phishing click means a headline, but downplaying real breaches is dangerous too. If you’ve ever had management panic over nothing (or shrug at everything), this episode is worth a listen.

🔮 Sneak Peek of the Next Episode

Next week, Paul Schnackenburg is back! We’re tackling Conditional Access in Entra ID. Expect practical advice, war stories, and why ignoring “break glass” accounts is like skydiving without a backup chute.

📝 From AndyOnTech and Project Runspace

Decision Making for SysAdmins and System Architects — by Eric Siron
Eric argues that the best SysAdmins aren’t the fastest typers or the fanciest scripters. They’re the ones who make the smartest calls before touching a config. This article covers mental models and structured thinking that help you avoid firefights before they start.

📚 Core Fundamentals

Microsoft AI-900: Azure AI Fundamentals
The AI buzz isn’t going away anytime soon. This entry-level cert introduces you to Azure’s AI services, responsible AI practices, and core concepts. Even if you never become an AI engineer, it gives you enough grounding to call BS in the next vendor pitch.

👥 Helpful Community Content

Azure Essentials Show: Building a Generative AI Center of Excellence
Thomas Maurer & Ben Brauer talk about scaling generative AI responsibly inside an org: governance, best practices, and how not to let “AI everywhere” turn into “AI mess everywhere.”

Tony Redmond: PowerShell for M365 Retention Policies
A deep dive into using PowerShell to wrangle retention policies across tenants. It’s practical scripting advice for anyone tired of clicking around the compliance center UI.

Tytus Kurek: How to Set Up an OpenStack POC
Thinking about private cloud or just want to kick the tires on OpenStack? This guide shows how to spin up a POC without dedicating a month of your life to it.

Reddit: Pausing Your Tech Career for Family
SysAdmins candidly discuss stepping away for family reasons and then re-entering tech. It’s raw, thoughtful, and a reminder that the career ladder doesn’t always go straight up.

🏢 Other SysAdmin Content from Vendors & Official Pubs

Microsoft: Limiting .onmicrosoft.com Email Sending
Starting soon, .onmicrosoft.com domains won’t be usable for outbound email. The intent is to cut off a common abuse vector used in phishing and spoofing. Check if you’ve got any workflows still tied to this before they silently break.

Experian: How to Request a Credit Freeze for a Minor
With reports of the entire SSA database floating around, identity theft risk just went way up. Freezing your own credit is easy; freezing your kids’ credit takes more paperwork but may be just as critical.

🔒 Security Headlines for SysAdmins

Cloudflare Impacted by Salesloft Drift Supply Chain Attack
Even big security vendors aren’t immune to supply chain risk. Cloudflare confirmed exposure via the Salesloft Drift compromise. If you integrate with SaaS platforms, treat vendor risk management as a core control, not an afterthought.

Microsoft on Storm-0501 Ransomware TTPs
Microsoft breaks down how Storm-0501 is abusing cloud services to launch ransomware campaigns. It’s a reminder that attackers are moving where the workloads are, and “the cloud” doesn’t magically protect you.

🛠 Tool of the Week

Pi-Hole
A local DNS sinkhole that blocks ads, trackers, and shady domains network-wide. It’s great for home labs and SMBs alike, and the dashboard gives you eye-opening insight into just how much garbage your devices try to connect to daily.

⚡ Quick Win of the Week

Audit your tenant for auto-forwarding inbox rules. They’re still one of the cheapest and dirtiest tricks attackers (or malicious insiders) use to exfiltrate data. In most M365 environments, you can surface these with a quick PowerShell one-liner and put a guardrail in place before it becomes a security incident.

🕹️ Fun Retro SysAdmin Fact

In the 1980s, 5-10 MB hard drives would be sold for up too $3,000. That’s roughly $300 per megabyte. Today, you can buy a 1 TB SSD for less than lunch money. Not a bad return on progress.

☕ Wrap-Up

That’s a wrap for this week! As always, you can find all the podcasts, newsletters, and resources over at SysAdminWeekly.com.

Go forth, patch responsibly, and don’t forget to hydrate. See you mid-week next Wednesday!

· Broadcom + VMware = SysAdmin headaches (and maybe a few choice swear words).

· Revisiting Hyper-V management drama. Which tool should you use, when, and why it matters.

· Next episode of the podcast: redefining “Compromise” in security (spoiler: it’s not just “the attacker owns everything on your network”).

· Stress testing Linux makes a comeback, because benchmarks matter.

· Core Fundamentals: Getting started with Entra ID (because it underpins literally everything in Microsoft’s cloud).

· Community gold: break glass accounts with TAP/FIDO2, AI thought leader lists, open source LLMs in enterprises, and benchmarking GPUs on Linux.

· Vendor news: Ubuntu Pro patches now surface in Azure Update Manager.

· Security news regarding RDP and Malicious Android Apps

· Tool of the Week: yabai, the MacOS tiling window manager you didn’t know you needed.

· Quick win + retro fact to close us out with a grin.

This Week’s Insight(s) from Andy

If there’s a theme this week, it’s “control slipping away”.

Broadcom’s VMware changes are forcing IT teams to rethink strategies they thought were stable. Microsoft is shifting how we handle identity and access in Entra ID (hello, break glass MFA requirements). Even Linux SysAdmins are talking benchmarking and stress testing, because performance drivers can make or break certain deployments.

The pattern? SysAdmins are being asked to manage less certainty with more responsibility. Vendors keep moving the goalposts, and we’ve got to adapt… with the right tools, smarter fundamentals, and a good dose of community wisdom to keep us sane.

I think a core theme throughout my (now) 23 year career in tech is the fact that long-standing “best practices” have a habit of changing, whether we want them to or not. An example….

I remember the days of centralized end-user computing. There was a time that Citrix was the “top dog” in this area. I remember deploying Metaframe XP and presentation server 4 and having a high degree of success across multiple environments. From there, Citrix starting becoming cost prohibitive (not unlike VMware now!), and many organizations moved to RDS (Remote Desktop Services) in Windows Server. After many of those deployments and ongoing support, admins learned very quickly that multiple users leveraging the same OS instance created some…. Challenges.

As a result the industry then shifted to VMware View…. Which then rebranded to “Horizon View”….. and….. and…. and…. you see what the theme is here. SysAdmins have LONG had to roll with changes in the industry and within specific technology areas (centralized end-user computing in this example). Adaptation, questioning the status-quo, patience (to resist changing too quickly), and staying up to date on best practices are all skills that will prepare SysAdmins for this sad-reality of the tech space. Remember, best practices are less ‘rules carved in stone’ and more like ‘guidelines’ scribbled on a whiteboard that vendors keep erasing when you turn your back.

And now…. Back to our regularly scheduled programming

Latest on the SysAdmin Weekly Podcast

🎙️ WHAT THE HELL DID BROADCOM DO TO VMWARE?! (Yes… all-caps are warranted)

Broadcom’s VMware acquisition fallout is in full swing. Licensing changes, customer pushback, and SysAdmins left holding the bag. Andy and Eric break down what’s changing, why it matters, and how to plan your next move if VMware is in your stack.

In Case You Missed It on SysAdmin Weekly

🎙️ Hyper-V Management Drama

Missed this one thanks to last week’s special Thursday edition (life prevented Andy from our usual FULL newsletter publication)? In this episode, Andy and Eric dive into the messy world of Hyper-V management. Mainly, what tools exist, when you should use them, and how to keep from losing your mind when Microsoft gives you four overlapping options.

Sneak Peek of the Next Episode

Next up later this week: “What is the definition of the word Compromise?”

We’ll be unpacking what compromise really means in a cybersecurity context. Spoiler: it’s way more nuanced than “someone got popped.” Expect a mix of philosophy, practical detection strategies, and a reality check for anyone tossing the term around without thinking.

From AndyOnTech and Project Runspace

📖 How to Stress Test Linux

We covered this one before, but worth a revisit with the benchmarking discussions going around the community this week. If you’re deploying workloads and want to test your Linux box under pressure, here’s a step-by-step breakdown on how to do it.

Core Fundamentals

📖 Getting Started with Microsoft Entra ID

Entra ID isn’t just “Azure AD with a new name”, it’s the backbone of Microsoft’s entire cloud identity stack. This Microsoft Learn documentation gives you the fundamentals you need to start strong.

Helpful Community Content

🛡️ Break Glass Accounts and Azure AD Security Defaults — Jan Bakker explains how to align Microsoft’s required MFA for critical management portals stance with the real-world need for bypassed break glass accounts. Solution: Temporary Access Passes + FIDO2 keys. Essential read if you’re setting these up (like Andy was recently).

🧠 Expert Circle – AI Thought Leaders — Sai Turlapati has done the community a solid with this list of AI experts, bloggers, and Microsoft tech leaders to follow. Great for curating your feed without doomscrolling LinkedIn randomness.

💡 What Open Source Models Mean for Enterprises — Barry Briggs breaks down the business implications of open source LLMs and how they’ll change enterprise strategy.

🖥️ Benchmarking 3D Graphics Cards and Drivers on Linux — Emmanuel Kasprzyk gets nerdy on how to properly benchmark GPU performance under Linux. Pair this with AndyOnTech’s stress test guide (above) for full effect.

Other SysAdmin Content from Vendors and Official Publications

🐧 Ubuntu on Azure: Pro Patches Now Surface in Update Manager — If you’re running Ubuntu on Azure, Update Manager will now surface missing Ubuntu Pro patches. One less place to miss something important.

Security Headlines for SysAdmins

Surge in Coordinated Scans Target Microsoft RDP Auth Servers

Nearly 2,000 IPs were caught hammering Microsoft RDP Web Access portals in a coordinated campaign. This is way above the usual background noise. Attackers are probing for timing flaws that leak valid usernames, with most traffic flagged as malicious and coming from Brazil.

👉 Lock down RDP: enforce MFA, restrict exposure, and hide it behind VPNs or conditional access. (please?)

Read more

Malicious Android Apps with 19M Installs Removed

Google yanked 77 malicious Android apps (19M+ installs) from Play, loaded with adware and banking trojans like Joker and Anatsa. Anatsa now targets 800+ financial apps, using sneaky payload delivery and Accessibility abuse.

👉 SysAdmins: remind users to enable Play Protect, avoid shady publishers, and scrutinize app permissions.

Read more

Tool of the Week

🪟 yabai – Tiling Window Manager for macOS

This one’s just… chef’s kiss. A tiling window manager for macOS that finally gives you power-user control over your desktop. If you’ve ever envied i3 on Linux, but were stuck on a Mac, this tool’s for you.

Quick Win of the Week

Set up a break glass account checklist in your documentation.

It takes 5 minutes to write down:
- Account names
- Stored credentials location
- Authentication method (TAP/FIDO2)
- When they were last tested

Future you (and your boss) will thank you when chaos strikes.

Fun Retro SysAdmin Fact

The first ever “patch Tuesday”… wasn’t on a Tuesday.

Microsoft’s original move toward scheduled patching in 2003 dropped on a Wednesday (gasp!) The “Patch Tuesday” brand didn’t get locked in until later, once sysadmins around the world begged for a predictable cadence.

☕ Wrap-Up

Another week, another reminder that sysadmins live in constant flux. Whether it’s Broadcom throwing curveballs, Microsoft rebranding identity yet again, or Linux folks benchmarking their way to glory.

Drink your coffee, give your dog (or cat?) a good pat on the head for stress-relief, and remember: control may slip, but your SysAdmin skills are the anchor.

Rather than leave your inbox empty, I wanted to pull together something useful: a curated roundup of every Hyper-V episode we’ve done on the SysAdmin Weekly Podcast. If you’re considering a move to Hyper-V (or just trialing it because Broadcom keeps lighting VMware on fire), this is your crash course.

Regularly scheduled programming returns next Tuesday, bright and early.

🎙 SysAdmin Weekly Hyper-V Episode Extravaganza

🔧 Ep. 017 – Hyper-V Management Drama and What Tool to Use When
Watch on YouTube
Hyper-V Manager? Windows Admin Center? Failover Cluster Manager? SCVMM? PowerShell? Azure Arc? Eric and I untangle the mess of management tools and explain when to use what (and when to just pour a drink instead).

🏰 Ep. 013 – Hyper-V in the Domain? Yea or Nea?
Watch on YouTube
Should Hyper-V hosts be domain joined, or live in a workgroup? This debate has been raging for years. Spoiler: Eric and I don’t exactly sit on the fence.

💸 Ep. 003 – Hyper-V vs. VMware: Licensing
Watch on YouTube
Licensing fun! (Yes, that was sarcasm.) But seriously, we break down the differences in software licensing between VMware and Hyper-V. If you’re running the math right now, this one’s for you.

✨ Ep. 002 – What’s New with Hyper-V in Windows Server 2025?
Watch on YouTube
From shiny new features to under-the-hood changes, Eric and I cover what’s actually useful in the Windows Server 2025 release.

☠️ Ep. 000 – No! Hyper-V is NOT Dead!
Watch on YouTube
Our very first episode of SysAdmin Weekly (hence the 000 episode number!), where we kicked things off by dispelling the “Hyper-V is abandonware” myth. Spoiler: it’s still a critical part of Microsoft’s strategy.

🌐 Ep. 001 – What is Azure Local?
Watch on YouTube
Not strictly Hyper-V, but deeply related. We peel back the layers on Azure Local, and and why Hyper-V is at the heart of it.

🗳️ Poll: What Topics Do You Want More Of?

SysAdmin Weekly works best when it’s a two-way conversation. Both the podcast and this newsletter are here for the community, so we want your input.

You can also reach us directly at contact@sysadminweekly.com. If you have direct questions, or funny stories, or show feedback, feel free to reach out! Also, all show links live at sysadminweekly.com.

Wrap-Up

That’s it for this special drop. Thanks for rolling with the schedule hiccup folks! We’ll be back with our regularly scheduled programming on the normal Tuesday morning cadence next week. Until then, may your patches apply cleanly and your VMs never bluescreen.

--Andy