• Paul Schnackenburg says AI feels like the early PC days. For me it’s the ESX 3 era all over again, and the best time to learn is now, while everybody is still figuring it out together.

• AI will hurt some people in this profession. I just don’t think it’ll be as bad as the doom narrative says, because SysAdmins already know the systems and the business, and that’s the trusted advisor seat.

• Episode 059: remote access is mostly risk assessment, and the protocol is the least interesting decision in it.

• Community Signal: Jeff Hicks on AI as a co-pilot, and a PowerShell agent loop that prompts before Format-Volume only by accident.

• Tool of the Week: ProxLB, DRS-style rebalancing for Proxmox, and where it still fits now that 9.2 balances HA guests natively.

• Quick Win: find every credential your AI tools are holding, then take one away.

From the Console

I have to preface this write-up with the statement that I do love working with tech every day and I’m wildly fortunate to be able to do that. That said… it’s been… a minute since any new tech REALLY hooked me. Agentic AI, AI harnesses, and actually seeing the power of AI outputs for myself changed that.

I had mentioned this while recording the upcoming podcast episode 060 with Paul Schnackenburg and he mentioned something that stuck with me the last few days. Paul stated that many SysAdmins seem to be dealing with the same experience right now. He stated that it reminds him of the early days of PC computing in that everything was new and people were experimenting with hardware and operating systems. That was true for me as well. I recall tinkering (to my parents’ dismay) with our 486 and figuring out how to reallocate memory so that I could play TIE Fighter and Dark Forces amongst other things during the DOS \ Windows 3.1 days.

As fond of those days as I am, however, the era of computing that said experimentation really reminds me of personally is the early days of virtualization. I remember looking over a senior administrator’s shoulder as they grew a virtual disk live…..while the machine was running. I then discovered vMotion (VM mobility between physical nodes) and DRS (Distributed Resource Scheduler) which would keep nodes load balanced. I remember the realization that updates to entire hypervisor clusters could be done with ZERO downtime if orchestrated properly.

Those early virtualization experiences were all from the ESX 3 era (think 2006-2009ish). I remember a lot of personal growth during that time frame, but I also remember that there was A LOT of shared learning happening as well. It was NOT uncommon for one of us Admins in the MSP engineering room to discover something new for the first time and for the rest of the team to gather around and excitedly “nerd-out” about how cool the discovered feature was and how it could be incorporated into existing operations.

That all brings me back to today and how the new AI craze has its hooks into me in much the same way. I know there are a lot of folks out there worried about the impact AI will have on society (more on that later).

The fact is: AI is here to stay. That genie isn’t going back in the bottle.

If you’re not learning how AI technologies can be used within your area of expertise, you’re doing yourself a disservice. This period of emerging AI technology is the BEST time to be learning about AI when you can learn and experiment alongside the rest of the IT community. Not only will you learn more, you’ll also be part of that community voice and help steer the use of AI within the SysAdmin community.

And now… back to our regularly scheduled programming.

Recently on the SysAdmin Weekly Podcast

Episode: How Should SysAdmins Handle Remote Access in 2026? (Episode 059) Topic: Eric and I take on the piece we deliberately cut from the tools episode, and most of it turns out to be about how long a path into your estate stays open. The protocol is the small part.

Why this one is a good listen:

• Eric lands the sharpest reframe about half an hour in: what overlay networks and their cousins buy you is a move from one-sided authentication to mutual authentication. A management port can still be reachable, and what “open” means changes completely once both ends have to prove who they are.

• The RMM stretch is the one for MSP folks. One agent on every endpoint, all of them reporting to one console, makes that console the most valuable thing an attacker can take from you. It leads to the two questions worth asking before any remote tool goes in: do we need remote management at all, and do we need it all the time?

• The back half turns the camera around. An attacker’s first move after a foothold is an outbound path (cloudflared is the example on air), and most SMB and mid-size shops are not watching outbound traffic closely enough to notice. Then the 3am phone call, where just-in-time access holds up right until the thing is down and nobody is awake to approve your elevation.

Watch on YouTube

Listen on Spotify

The Take

My recent obsession with Agentic AI workloads ultimately led to this week’s “take”. It could also be my affinity throughout my career for automation as well. I mean…. my very first IT mentor DID convey to me that:

“Andy, the key to IT is Laziness. If something MUST be done more than once, then it MUST be automated”.

That’s stuck with me throughout my career. Fast-forward now to the AI era and the capabilities for “automation” via Agentic AI are astronomical.

Let me be clear before I go further on this one. I have NO DOUBT that there are people out there that have been negatively impacted or will eventually be negatively impacted by emerging AI technologies. Eric and I discussed as much alongside our thoughts on the AI Doom Narrative back on episode 048 of the podcast. We also discussed in that same episode the numbers from the World Economic Forum in regards to job gains (yes, gains) between now and 2030.

Here’s my take for this week’s issue of the newsletter. Again I have no doubt that AI will impact some people negatively in the SysAdmin profession, but I don’t think it’s going to be as bad as the ongoing AI doom narrative would have you think.

SysAdmins, as a profession, are well positioned to help organizations reap the benefits of AI.

Think about it:

• SysAdmins have deep knowledge of existing technology systems within their given environments.

• SysAdmins understand many of the business processes and intended business outcomes for the organizations they are part of.

• SysAdmins already have the technical background to uptake AI capabilities at an accelerated pace vs. your average business office worker.

• For historical context, the VMware ESX and virtualization stuff was “going to cost IT jobs” as well. It didn’t, it just changed the profession, just like AI will.

With all this in mind, I see the future SysAdmin as falling increasingly into the “Trusted Advisor” role. In this role SysAdmins can advocate for the best use of AI within their organizations leading to increased automation, cost-savings, and productivity gains. All of those things make you a hero as far as management is concerned (typically).

When looking at traditional IT work specifically, AI is likely only going to help us in the long run. Already, AI has the ability to interact with infrastructure systems as needed. When scoped and sandboxed PROPERLY, AI can be a valuable SysAdmin assistant. I foresee that many SysAdmins will be managing and deploying more than ever before because AI will allow us all to go faster and further.

That all said, I’m very aware of the dangers posed by AI. We MUST tread carefully as an industry as we incorporate these new technologies into existing IT stacks. For example, in the last episode of the podcast (listed above), Eric and I talked about remote access. AI is practically another remote access point into your environment. Treat it as such. Further examples aside though, being on the front lines of tech, we SysAdmins are best positioned to be enabled and uplifted by AI as opposed to being cut and thrown into a RIF (reduction in force).

I’d love to know what you all think in the comments below!

Community Signal

Jeff Hicks (long-time Microsoft MVP) on The PowerShell Podcast - “PowerShell Wisdom from 35 Years in the Trenches” - April 13, 2026, 45 minutes. Hicks has been teaching PowerShell since the beginning, and the show notes sum up his position on AI as “a co-pilot, not a replacement.” He uses it to get past a specific technical hurdle rather than to hand him a finished script. The worry he raises is the useful part: a generation that skips the foundational learning can’t recognize when the AI gets it wrong. His advice for anyone starting out is the old advice, and it holds up: objects, the pipeline, and managing at scale first, specific modules later. Source note: the podcast is produced by PDQ, which sells endpoint management software. Hicks is independent, and the episode is billed as a conversation about community and craft.

Doug Finke (16-time Microsoft MVP) - “Building a PowerShell AI Agent: From Scratch to Claude Code” - January 10, 2026. If “agent” still sounds like magic, read this. Finke, who created the ImportExcel module, builds one in PowerShell in three passes: a script that asks the model for one command and waits for your y/n, a loop that keeps going until the model says it’s done, and then structured tools for reading files, writing files, and running commands.

Tool of the Week

ProxLB - An open source, GPL-3.0 load balancer for Proxmox VE clusters that watches node utilization and live-migrates VMs and containers to even it out, which is DRS-style rebalancing for the hypervisor a lot of you moved to.

It pulls CPU, memory, and local disk metrics through the Proxmox API, and when the gap between your busiest and quietest nodes crosses a threshold you set (the project calls it “balanciness”), it migrates guests until the gap closes. You can balance on used resources, assigned resources, or pressure stall information. Maintenance mode drains a node before you patch it and can run on a schedule. Affinity and anti-affinity rules keep your two domain controllers off the same host.

Honest scope, and the ground just moved under this one. Proxmox VE 9.2 shipped on May 21, 2026 with native dynamic load balancing, but only for HA-managed guests. ProxLB’s case now is everything outside that line, mainly guests that are not in the HA stack and clusters still on 7.x or 8.x. The README has a side-by-side comparison against the native balancer that is worth reading, but the ProxLB project wrote it and the scoring reads that way. The native feature is supported by Proxmox and upgrades with the platform. ProxLB is a daemon you validate yourself after every Proxmox upgrade.

Two more things before you install it. The original author, gyptazy, handed the project to his employer credativ GmbH in January 2026 after people raised concerns about a one-person project’s long-term maintainability, so the repository has moved, and credativ offers paid enterprise support for it. Oh…. and don’t point two schedulers at the same guests on a hunch. The README still carries an older warning against running ProxLB alongside HA groups, while the 2.2.0 release in June added HA mode support.

Read the documentation for the version you install, and exclude anything the balancer should never touch before you turn on the daemon.

Quick Win of the Week

Find every credential your AI tools are holding, then take one away.

Pick the machine where you use an AI coding agent or assistant the most, and open the shell it runs in. Anything that session is logged into, the agent can use. gh auth status shows the GitHub account and its token scopes, and az account show shows which Azure subscription you’d be pointing it at.

Then read the MCP config files, the ones people paste tokens into late at night to get a server working. Claude Code keeps user and local scope servers in ~/.claude.json and project scope servers in .mcp.json at the repo root, which gets committed along with everything else. VSCode keeps workspace servers in .vscode/mcp.json. Any bearer token or API key sitting there in plain text goes on the list. Both tools’ documentation tells you to use variables instead of hardcoding it.

For each entry, write down the tool, the credential, what it can reach, and when it expires. Then fix one: swap a classic GitHub token with full repo scope for a fine-grained token limited to the repositories the agent actually works in, with an expiration date on it.

Nothing revokes itself unless you build it that way. That line was about SSH keys in episode 059, and it applies just as well to whatever you hand the new tools.

Fun Retro SysAdmin Fact

The personal computer era everybody gets nostalgic about started in a garage: on March 5, 1975, the Homebrew Computer Club held its first meeting in Gordon French’s garage in Menlo Park, gathered around the first MITS Altair 8800 to reach the area (a review unit sent to People’s Computer Company), and Steve Wozniak later credited that meeting as the inspiration to design the Apple I.

Until Next Week

Enjoy the new tool, then go find out what it’s logged into.

Stay Frosty,

Andy

SysAdmin Weekly

• Five weeks on the road out of six, and the laptop never noticed it left the house.

• Identities and ACLs are your firewall now, and the change control discipline never followed them there.

• Episode 057: the air gap that is not an air gap, and every hole you opened for a good reason just to manage the thing.

• Claude Code running on a lab VM, driven from a phone over /rc, and the access questions worth settling first.

• Tool of the Week: Headscale, for when you want the mesh without the coordination plane you do not own.

• Quick Win: get off your own network and walk the path you would actually use at 2am.

From the Console

It hit me at a couple of different points over the last few weeks:

• From an Airport gate in Michigan (literally 20 minutes from my house) where my flight was drip scheduled for 12 hours before being canceled

• From a hotel room at New York, New York in Vegas with a literal rollercoaster in view out my hotel window.

• From a bar at a PGA resort in Frisco, Texas.

• From an Airbnb that reminded me of a Hobbit hole in Minneapolis.

• From a floating hotel on the Thames in London.

• From 20 thousand feet on an American Airlines flight to Dallas.

For the most part during all that travel my laptop never really thought that it existed anywhere other than my desk at home. The exception, of course, were the occasional captive portal (looking at you American Airlines) that required local-only routing. Other than that, my system blissfully thought it was sitting on my desk at home. I could access “local” systems, M365 saw me as connecting from the same GEO, and I could also troubleshoot family tech issues while not being in the room. At one point I looked up and thought to myself “This has been effortless for weeks….”. The tool had become so easy, transparent, and had just gotten out of the way to the point where I just STOPPED thinking about it…. which is what a good tool does. It enables you, without becoming all about itself.

Maybe this wouldn’t have been noteworthy for those SysAdmins that have entered the IT industry during the cloud era, but I remember a time where persistent, secure, and transparent remote access was a legit feat of engineering. Either:

1. The user would have to manually establish connectivity upon startup

2. 3389 was just left open to a “jump box” in the target environment (YUCK!)

3. The user experience was impacted in some usability way

With modern hybrid cloud technologies, work from anywhere is easier than ever and completely transparent. Yes, we’ve had VPNs for a long time, but overlay networks, like Tailscale, are on a whole new level. Never has it been easier to connect multiple geographically separated systems / networks than it is now. For example, sitting at a gate at the airport… with an overlay network, I can securely access my home network, my production management network, lab resources, as well as a hosted web server that is locked behind the vendor providing it. All of it just effortless, and secured via ACLs, routing options, and filtration.

While I’m obviously a HUGE fan of this configuration, two things do come to mind.

1. Just because your current configuration has been in place and working for a period of time, it doesn’t mean that there isn’t a new solution out there that can do it better. Look up once in a while and see if the status quo is still the configuration that makes sense.

2. As good as new technologies are, don’t become OVER-dependent on them. You may not always have an overlay network available to abstract and automagically configure networking and routing. Those basic network skills are still important and will be needed when tools (like Tailscale) fail for any given number of reasons.

And now… back to our regularly scheduled programming

Recently on the SysAdmin Weekly Podcast

Episode: Is Your Air Gap Actually an Air Gap? Port 22 Says No (Episode 057) Topic: Almost nothing called an air gap is one. Eric and I walk the management plane hole by hole, and we land somewhere more useful than telling you that you are doing it wrong.

Why this one is a good listen:

• We go through the holes in the order they get opened, and the list is uncomfortable because none of them were mistakes: SSH and RDP so somebody can patch the thing, SMB because files have to move, DNS and NTP because nothing works without them, then license activation and telemetry because a vendor said so. Every one of those was opened by a competent person with a good reason on an ordinary Tuesday.

• The iDRAC, iLO, and IPMI stretch is the one to forward to your team. Those boards get parked on a management VLAN, keep whatever firmware shipped on them, and then nobody touches them again, which quietly makes the out-of-band path the least defended road into the most privileged place you own. There is also a detour about printers that we both stand behind.

• The close is the part that changed how I use the word. A real air gap is possible and it costs a great deal, most of it in operational pain rather than dollars. What actually hurts people is the term: call something an air gap and the conversation ends, and nobody re-verifies a promise. Say what you have, then defend that.

Watch on YouTube

Listen on Spotify

The Take

While we’re speaking about painless remote network access in this issue I think it’s worth highlighting something I’ve noticed over the last few years in forum posts, conference conversations, and on social media. Identities (and ACLs) ARE your firewall now. It used to be we only had to worry about the 4 walls of our building. We’d put a firewall on the entry point and call it good, but with the cloud era and work from everywhere, that model doesn’t work so well anymore. So, that’s where ACLs, Conditional Access, Cloudflare Access rules…etc come in.

With that in mind then, I’ll pose a question: Do you remember the last time your ACLs were audited? Were those recent Conditional Access changes run through change control? As an industry we got REALLY good (mostly) at including even the simplest firewall changes in the change control process. I’ve found that the practice seems to be slipping for identity in the cloud era. Add on the fact that there are a myriad of ways SysAdmins can manage these various settings and change control becomes even MORE important in today’s world.

Why did this happen? I’m not convinced it’s laziness. It’s easy to include a firewall in that process as someone has to order that box, rack it, stack it, configure it…etc. The security model for cloud services doesn’t lend itself well to that mental model and as such, securing those services often becomes an afterthought and they get left out of change control processes.

Something to consider: Do you have any identity management workflows that go un-audited? Anything that is left out of change control? If there are, it’s time to take stock of them and start putting tools/processes in place to address those shortfalls.

None of it is exotic:

• Put the policy in version control. Tailnet ACLs, Conditional Access exports, access rules for whatever tunnel you run. If it decides who reaches what, it belongs in a repo.

• Put a reviewer on the pull request. You wouldn’t let a firewall rule go in unread, and this is the same thing wearing different clothes.

• Let the diff be the change record. It’s already written, already timestamped, and it already says who approved it.

• Write down every console that can change these settings. That myriad (there I go using that word again!) of management options is exactly why this slips, and let’s face it... you can’t audit a surface you don’t list.

What I Learned in AI this Week

A quick one here this week

If you haven’t looked at the /rc option in Claude Code, I HIGHLY recommend you take a look and determine ways you can incorporate it into your workflow. With all the travel I’ve been doing lately I made heavy use of this feature. My usage breaks down pretty simply:

• I have a VM running in my lab that acts as an agent “worker” machine.

• Said VM is also used to run Claude Code against locally synced git repos when I’m remote

• I launch Claude Code using Screen so it survives ssh disconnections

• Enabled /rc in the Claude Code Session

• Connect using the Claude app on my phone

It seems a simple thing, but the ability to issue long running jobs and prompts remotely while I’m on the road is immensely valuable. If you also make sure that the bash session you launched code with is logged into GitHub’s CLI, Claude can even push/pull and open PRs.

That all said, there are security considerations here. Make sure your Claude account is not easily breached. Also consider the level of GitHub access you’re comfortable giving Claude. In my case, I don’t really have anything stored in GitHub that I would care about Anthropic seeing. Hardly an exhaustive list, but something to think about as you’re involving more AI tools in your standard workflow.

Community Signal

Brandon Lee - “This Made My Mini PC Home Lab Feel Enterprise Grade: Intel vPro with AMT” - Published February 23, 2026. Out-of-band management is the layer that decides whether “I can work from anywhere” survives contact with a node that has stopped answering, and Lee walks the whole thing on hardware normal people actually own: MEBx configuration on a Minisforum MS-01, MeshCommander in Docker, remote KVM and power control and BIOS access when the OS is gone. The part worth reading twice is the operational detail he did not have to include, like the AMT NIC not supporting VLAN tagging and the HDMI dummy plug trick for keeping video alive. He is also blunt about the risk: treat it like iDRAC in production, put it on a dedicated VLAN, and never let it see the internet. Honest scope, this is mini-PC specific, so the exact steps are for that class of hardware rather than your rack.

Tool of the Week

Headscale - An open source, BSD-3 licensed reimplementation of the Tailscale control server, so you can run the coordination plane for your own mesh instead of renting somebody else’s.

The tool I said got out of the way up top does that partly because somebody else runs the control plane. This is the version where that somebody is you.

The clients stay the official ones. What changes is who operates the piece that hands out keys, publishes the network map, and enforces your ACLs, which for a lot of shops is the exact objection that stopped the conversation the first time. It covers the features people actually use day to day, including MagicDNS, subnet routers, and exit nodes, and it is a single binary with a config file rather than a platform.

Honest scope, and this one matters more than usual. The project states its own goal plainly: a server suitable for self-hosters, hobbyists, and small open source organizations, built around a single tailnet. It is not trying to be a multi-tenant enterprise control plane, the maintainers explicitly say they do not support or encourage running it behind reverse proxies or in containers, and the project is not affiliated with Tailscale Inc. There is also a trade you should name out loud before you make it: the whole appeal of the managed service was that somebody else kept the coordination plane running, and self-hosting hands you that availability problem. Think through what happens to node enrollment and policy changes on the day your Headscale box is the thing that is down.

It is also still pre-1.0 and it means it. Version 0.29 changed what a wildcard resolves to in policy: * in ACL sources and destinations now covers the CGNAT and ULA ranges rather than every address. Upgrade without reading that line and your ACL file means something different than it did the day you wrote it, and nothing is going to tell you. That is this issue’s Take, delivered by the tool itself.

One more thing before you bolt a UI onto it. Headscale ships without one, so the docs point you at eleven community-built options, and the most popular of those is where this stack’s recent problem turned up: Headplane, CVE-2026-46484, scored 8.1 and patched in May 2026, which let an authenticated user rename or expire any node or user in the tailnet through a path traversal in the rename call. Headscale itself has had exactly one advisory in its entire history, back in 2023. The convenience layer is the part that needed the patch, which is worth remembering when you pick what else to bolt on.

Quick Win of the Week

Get off your own network and walk the path you would actually use at 2am.

Tether to your phone. Not your home wifi, not the office, not the coffee shop you have used a hundred times. Pick one machine you would have to fix if it stopped answering tonight, and go all the way to a working console on it. Not SSH into a jump box you already trust, the actual box. If getting there requires out-of-band, use out-of-band, because that is the path the outage will force you onto anyway.

Write down every step that fails, prompts you for something you do not have on you, or needs a second device you left at home. Time-box the whole thing to an hour.

That list is your real remote capability. The one in the policy document is a different document.

Fun Retro SysAdmin Fact

SSH exists because somebody got sniffed: in 1995 Tatu Ylönen, then a researcher at Helsinki University of Technology, wrote the first version in response to a password-sniffing attack on the university network, released it as freeware that July, and watched it reach roughly 20,000 users across fifty countries before the year was out, which means the tool that eventually untethered every SysAdmin from the server room was built as incident response rather than as a convenience feature.

Until Next Week

Your estate does not care where you are sitting, right up until the one path you never tested is the only one left.

Stay Frosty,

Andy

SysAdmin Weekly

• Spoke at Black Hat for the first time, spent the run-up convinced the room would find my demo old news, and got a lesson in imposter syndrome instead.

• Old and battle-tested is exactly why people stop looking, and Active Directory is the proof: 25+ years in, researchers are still pulling full domain takeover out of Kerberos.

• Two AD vulnerabilities worth knowing by name: KerberLoss and ResetNightmare. Both already patched. Whether your domain controllers took those patches is a separate question.

• Episode 056: Andy and Eric Siron on how they actually got into this work, the advice that held up, and the advice that did not.

• RC4 enforcement went permanent with the July update, and the domain-wide rollback key went with it.

• Quick Win: hunt KDCSVC enforcement errors on every DC, plus the years Kerberos spent classified as auxiliary military equipment on the US Munitions List.

From the Console

I recently was able to mark something off of my bucket list…. kind of. Near the beginning of August, I had the pleasure of speaking at Black Hat for the first time. Ideally, this would have been via a CFP (call for presenters) selection, but it was part of a sponsorship deal. Still… speaking at Black Hat in a sponsored showfloor session was cool with me as well.

As regular podcast listeners will know, I speak at cybersecurity and MSP shows quite regularly and as covered in episode 049 of the podcast I spoke at Infosecurity Europe 2026 earlier in the year about how threat actors leverage locally hosted LLMs (via tools like Ollama) to do automated spear phishing generation at scale. I ended up doing much the same topic, but with some core changes.

See, Black Hat is known for being highly technical, almost to the point of intimidation in my opinion as a speaker. I knew that for the average SysAdmin in the trenches, the Local LLM spear-phishing demo would be a bit shocking to see on stage. With the technical crowd at Black Hat… I wasn’t so sure. In fact, I was a bit nervous about that going in.

So what’d I do? I made the content more technically complex. Additional demo paths, alternate scenarios. MORE time spent in the CLI and VERY few slides. Even with all of that, I was, admittedly, still worried that the (now) known attack techniques attackers are running via local LLMs would be old news to the crowd. Pair that with the fact that many of the other show floor theater sessions were very sparsely attended and…. yeah… I was getting a bit nervous.

Despite all of that, I had excellent turnout for my technical session and was even surprised at the number of people taking pictures of the attack as it progressed with some even recording it. I don’t share all of this to brag….no, there is a clear lesson to be had here and it centers on imposter syndrome in our industry. Tech fosters a culture that makes people hesitate to share knowledge or ask questions out of fear that they SHOULD already have the answer, and will, thus be seen as dumb.

In hindsight, my reaction leading up to the session was imposter syndrome. I was doing it, and I was wrong.

The part I’ll leave you with today is some advice I received from a friend working in tech education MANY years ago and it holds true today:

Don’t be afraid to share your knowledge with others out of fear that they won’t find it interesting or that you think it’s already general knowledge and that you’re behind. Instead, remember that everyone is at different stages of their tech careers and either need the information that you’re willing to share for this stage of their professional lives, or maybe they gain a new viewpoint on something they already knew because of your unique delivery style or lived experience.

And now…. back to our regularly scheduled programming.

The latest on the SysAdmin Weekly Podcast

Episode: How People Actually Get Into SysAdmin Work (Episode 056) Topic: Eric and I trade full origin stories, then put the career advice we were handed on trial. Some of it held up for thirty years. Some of it was nonsense the whole time.

Why this one is a good listen:

• The nerd hour segment is the technical companion to what I wrote up top. It walks through the Black Hat demo itself: the local models doing the spear phishing generation, the second script that made every lure Microsoft 365 specific, and the pass through an HTML parser that turns raw model output into something that lands in an inbox looking like an ordinary SharePoint share notification.

• The advice audit is the part worth your time. “The key to IT is laziness” came from my first mentor, meant automate anything you do more than once, and has held up for thirty years. “Always,” “never,” and “well, this is a best practice” did not, and the reason is the useful part: chase most of those “best practices” back far enough and you find one person on the internet citing another person on the internet with no original source anywhere. Ask who decided, and ask who made them authoritative.

• Both of our first real disasters were data disasters, which turns out not to be a coincidence, and the stretch after that is the one I would point a newer admin at: normalizing “I don’t know” at every stage of a career rather than only once you have enough scar tissue to say it safely. Eric’s version of that is blunter than mine. It closes on whether cheap hardware and time to break things still exist as an entry path, and we land more optimistic than you might expect.

Watch on YouTube

Listen on Spotify

The Take

The tech industry moves fast. So much so that we often lump “old” and “legacy” software together. Both of those designations can byte (hah! see what I did there?) you if you’re not careful. Take “old” on-prem Active Directory for example. It’s been the bedrock of enterprise identity for 25+ years and, despite its age, is still widely used by SysAdmins around the world today.

An “old” but tried and tested piece of software like Active Directory is seen by many to have been battle-hardened over the years and while knowing that something has been hardened and secured via attrition is a good thing, it causes many to drop their guard. Many a time have I heard someone say things along the lines of:

“AD (Active Directory) has been around for AGES… surely they’ve found all the bugs and security issues by now”

Me denying that statement is NOT today’s “Take”, but I feel like my take for this week is that I think generalist SysAdmins have a tendency to drop their guard on older, tested, and “mature software”. I don’t think it’s from a lack of skill / knowledge, but more so the mere fact that most are so overworked with the day-to-day that the small respite from lightening the mental cybersecurity load is sorely needed and welcomed. I’m not necessarily saying that this is right or ok, but that I understand it, and would urge those admins to put systems in place to watch for vulnerabilities for ALL software in your environment, and not JUST the more modern software.

For an example of why, let’s go back to the recent Black Hat conference again. During my time at the event I was able to attend a few sessions, and one in particular caught my eye. Shai Laron from Semperis was giving a session talking about different ways that he and his team were able to abuse Kerberos tickets, mainly by fuzzing UPN / SPN calls using unicode characters.

Using this method, Shai and his team surfaced two separate vulnerabilities:

1. KerberLoss (CVE-2026-25177) - unicode in a Service Principal Name creates a name collision that AD’s uniqueness check does not catch. That gets you a denial of service against HOST-mapped services, SPN-jacking for constrained delegation abuse, and the ability to force any service in the forest to fall back to NTLM. Requires WriteSPN on any object. Patched March 10, 2026.

2. ResetNightmare (CVE-2026-27912) - set your own UPN to match another account’s sAMAccountName, request a ticket as them, and reset their password without ever knowing the old one. Any domain user can write their own UPN by default. This is the full domain takeover. Patched April 14, 2026.

NOTE: Both of the above CVEs have already been patched back in March and April of 2026.

Full writeup from Shai with the mechanics of both: Semperis - “Identity Crisis: Novel Vulnerabilities Leading to Kerberos Downgrade, DoS, and Full Domain Takeover”

In short, in 2026 Shai and his team were able to completely own an established AD domain by essentially confusing Kerberos.

Here is a system that has been battle-hardened over 25+ years in the industry, and people are STILL finding vulnerabilities and exploits for it today. So, that seemingly old and “bulletproof” piece of software within your environment? Don’t ignore it, because it leads to getting hit with an exploit when you least expect it.

Community Signal

João Ferreira (Microsoft MVP) - “Enforcement phase for Kerberos RC4 protections begins with the July 2026 Windows security update” - Published July 15, 2026, and the timing matters more than the length. Audit mode and the RC4DefaultDisablementPhase rollback key were both removed with the July update. Ferreira’s list is short and correct: confirm your service accounts speak AES, close out the RC4 dependencies you found during the audit phases, and go read your DC event logs before someone else finds them for you.

Jerry Devore (Microsoft, Core Infrastructure and Security Blog) - “Active Directory Hardening Series, Part 4: Enforcing AES for Kerberos” - Refreshed August 2, 2026, and it is the most practical thing written on msDS-SupportedEncryptionTypes anywhere. The trick worth stealing is using Event 4768 to find your RC4 holdouts, because the ticket encryption type field reflects the session key issued with the TGT, which tells you which clients can only do RC4 rather than which ones you assumed could. Devore also names the four usual suspects: RC4-only keytab files, non-Windows integrated devices, Windows boxes with AES turned off by GPO, and anything predating 2008. Fair warning on sourcing, this is Microsoft’s own blog rather than independent community work, but it is field-engineer material and not marketing.

Quick Win of the Week

Go read your domain controllers’ System event logs for KDCSVC events 203, 204, 208, and 209. Those are the enforcement-mode error variants, which means they are not warnings about something that might break later, they are records of authentication that already failed since the July update landed. Their warning-mode twins are 201, 202, and 205 through 207.

Get-WinEvent -FilterHashtable @{
LogName = ‘System’
ProviderName = ‘Microsoft-Windows-Kerberos-Key-Distribution-Center’
ID = 201,202,203,204,205,206,207,208,209
StartTime = (Get-Date).AddDays(-30)
} | Group-Object Id | Sort-Object Count -Descending

Run it on every DC, not the one you always log into. While you are in there, confirm the box actually took the March 10 and April 14, 2026 rollups, since those are the KerberLoss and ResetNightmare fixes and nothing above will tell you if it did not.

Fun Retro SysAdmin Fact

Because Kerberos 4 used DES, the United States classified it as auxiliary military equipment on the Munitions List and banned its export, so MIT shipped an international version with every encryption function and every call to one surgically removed, named “Bones,” until Eric Young at Bond University in Australia re-implemented DES back into it to produce “eBones,” which went on to become the ancestor of the Swedish KTH-KRB and Heimdal implementations.

Final Item Worth Your Time

Semperis - “Identity Crisis: Novel Vulnerabilities Leading to Kerberos Downgrade, DoS, and Full Domain Takeover” - Shai Laron’s writeup of the Black Hat research behind this issue’s Take, with the full mechanism for both KerberLoss (CVE-2026-25177) and ResetNightmare (CVE-2026-27912). It is a vendor blog and there is a product pitch in the detection section, so read it for the research and skip the rest; the CVEs and the patch dates stand on their own.

Until Next Week

Patch Tuesday is a date on a calendar, not evidence that anything actually happened. Make sure those patches get installed!

Stay Frosty,

Andy

SysAdmin Weekly

As regular followers of the show will know, we run the whole list of SysAdmin topics for the show including:

• Microsoft’s ecosytem (Especially Hyper-V)

• M365 and Azure

• Cybersecurity

• Some Linux / Open Source stuff

• Storage and Networking

• And of course… soft skills

Soft skills are often overlooked when viewed with a list of other SysAdmin skills, but they are some of the most important skills in any SysAdmin’s toolbelt. They help you deal with the day-to-day and stay sane and they also help you communicate to a board room and provide customer service effectively as well.

As such, I tasked Claude with parsing the full episode list of the show (all 56 episodes across our back catalog of 15 months!) and had it curate a list of soft-skill episodes that we’ve covered throughout the lifespan of the show, thus far.

If you have a SysAdmin soft-skill topic you’d like to see us cover, do let us know in the comments below!

Here is the list!

Getting In

The on-ramp questions. Worth reading if you are early, or if you are the one doing the hiring and wondering why the pipeline looks the way it does.

Episode 005 - Microsoft Certification in 2025: A Microsoft Certified Trainer’s Take - Paul Schnackenburg is an MCT, so this is the view from inside the machine: the shift from product exams to role-based ones, why the exam names confuse everyone, what applied skills actually prove, and whether the annual renewal treadmill earns its place in your calendar. Useful if you are deciding where a training budget goes. NOTE: this one is a bit dated as far as certs go, but i still contains a TON of useful information for those navigating the Microsoft certification world.

Episode 027 - Is University Worth It for Aspiring SysAdmins? - Clay Tamam is a recent IT graduate in the Netherlands who was in his first SysAdmin role when we recorded, so this is not two veterans speculating about kids these days. Where the degree held up, where it did not, and how certifications and a homelab stack against it. Includes his first real outage, which for once was not DNS.

Episode 033 - Why IT Job Postings Are Completely Broken - Andy solo, scrolling through real postings: unicorn requirements, role creep, and job descriptions that read like four jobs stapled together. Not a recruiter dunk session. The argument is that a posting is a small glimpse into the organization doing the posting, most of them are posting confusion, and both sides can do better.

Episode 056 - How People Actually Get Into SysAdmin Work - Published this week, so it is the newest thing on this list. Andy and Eric Siron trade full origin stories, from a 486 and a VIC-20 through first paid jobs neither of them was qualified for, and land on the same first catastrophe: data gone, no idea if it was coming back. Closes on whether the cheap-hardware-and-time on-ramp that produced both of them still exists in 2026.

The Craft Nobody Tests You On

There is no exam objective for any of this, which is exactly why it separates people.

Episode 009 - What Makes a Great SysAdmin? - The one that started this whole thread for us. Andy and Paul pull apart the technical skills from the traits that actually make someone good at this work: communication over compiling, curiosity as the real fuel, and yes, empathy. Also gets into gatekeeping, and why the best people in this field never stop learning even when they badly want to.

Episode 015 - The Art of Troubleshooting in Tech - Change one thing at a time. That is the entire discipline, and almost nobody manages it at 2am with a queue backing up behind them. Scoping a problem without panicking, when to escalate, surviving vendor support roulette, and how to stop a junior tech from chaotic-clicking their way into a production outage.

Episode 022 - Why IT Documentation Matters (Even if You Hate Writing It) - Writing is a soft skill and this episode treats it like one. War stories where future you had to clean up past you’s mess, plus the honest question of what makes documentation stay alive instead of quietly rotting in a wiki nobody has opened since the last migration.

Getting Heard

Being right is not the job. Being right where someone with budget can hear you is the job.

Episode 014 - How SysAdmins Can Prove Their Value (Before the Next Layoff) - The Visibility Paradox: the better you are at this, the less anyone notices you exist. This one is about fixing that deliberately. Quantifying wins, turning automation into an ROI number somebody upstairs will repeat, and asking who is in the room before the meeting starts.

Episode 050 - How Do You Run a Blameless Incident Postmortem? - A postmortem that ends with a name instead of a root cause wasted everyone’s time in the room. Blameless without sliding into unaccountable, separating root cause from contributing factors, and why the follow-through is the entire point. The discipline scales from sixty people in a war room down to you writing a summary for one nervous boss.

Episode 055 - How Do You Run IT With No Budget? - “No budget” is almost never no budget. It is no budget yet, and what closes the gap is your ability to attach a real dollar figure to the risk you are already carrying, because leadership does not act on fear but it does act on risk management. And if the answer is still no, get the decision in writing.

Staying in the Chair

The long game. Nobody warns you that lasting twenty years in this field is its own separate skill.

Episode 010 - Tips for Tech Conferences in 2025 - Conferences are expensive and most people get them wrong. The hallway track is worth more than the session schedule, the expo hall is not purely a capitalist trap, and speaking at an event is one of the better career moves available to you. Recorded while Andy was still fighting off the conference cold that proves he was there.

Episode 026 - Burnout in IT: Why So Many Tech Pros Are Struggling (and How to Cope) - The heaviest one we have recorded, and the one most worth forwarding to somebody. Imposter syndrome in a field that changes underneath you, always-on culture, notification overload, and what actually moves the needle: boundaries, peer support, and managers who build psychological safety rather than put it on a slide.

Episode 031 - How the IT Community Makes You a Better SysAdmin - Not the corporate version of the word. Mentors who turned up unannounced and changed a trajectory, MVP circles, open source projects that operate nothing like vendor ones, and the case that contributing small things builds a career faster than any credential on its own.

Episode 042 - Should SysAdmins Job Hop or Stay Put? - Four decades of combined experience arguing that the answer is neither one. Staying long term buys institutional depth, ownership, and skill calcification you will not notice happening. Hopping buys pay jumps, breadth, and a reputation you have to manage. Secret Option C is the actual episode.

Episode 053 - How to Survive as a Solo SysAdmin - A listener request. When you are the only person between a working business and total collapse, the job stops being about doing everything and starts being about deciding what not to do this week. Triage before projects, documentation as an insurance policy, buying time back through automation, and speaking business value instead of acronyms.

One Closing Note

Line these up and a pattern shows up that none of us planned.

Every single one of these episodes is about the same thing from a different angle: the gap between doing the work and being understood doing the work. Documentation is that gap in writing. The postmortem is that gap under pressure. The budget conversation is that gap with money attached. Burnout is what happens when the gap never closes and you keep absorbing the difference yourself.

The technical skills get you in the door. This is the set that decides how long you stay and what you get paid while you are there.

Stay Frosty,

Andy

SysAdmin Weekly

• Rebuilt the lab’s backup strategy on an ancient ReadyNAS and an Azure storage account, and the boring tool you know cold beats the clever one you don’t.

• Your backup target and your management console have the same failure mode: one of anything is the problem.

• Episode 054: Windows Admin Center is a good tool with an on-ramp that ate four hours of Andy’s time.

• Hot take: Hyper-V’s pile of “disparate” management tools is a feature, and a stronger management story than it seems on the surface.

• Apache Guacamole as a second road into the fleet, plus the RAID acronym’s quiet rebrand once vendors started charging for it.

From the Console

Regular listeners of the Podcast and those who keep up with this newsletter on a regular basis know that I’ve been juggling a bunch of changes in the homelab lately. That said, I’m hoping to put homelab talk on the back burner for this publication for at least a few editions after this week’s entry. However, my “sitting at the console” work for this last week (at least in my personal time) has focused heavily on backups for the home lab.

Yes, I agree…. backup / recovery is boring. It’s the plain wheat toast of the SysAdmin world, but all of us should be in agreement when I say: Backup is Critically important. If you don’t agree with me on that one… we need to have a chat BTW…

After ripping the KVM virtualization layer out of my lab and replacing it with Hyper-V the number of backup options available to me broadened. Yes, I was using Restic and Proxmox Backup Server successfully, and they are GOOD tools, but for my given setup, they were both taking too much time and effort to manage.

I replaced them with VM Backup from Hornetsecurity by Proofpoint (Not a product plug, just a statement of fact).

Full disclosure: that tool and I go WAAAY back because I’ve worked for ALL of the vendors that have held that IP. I was at Altaro, the company that originally built VM Backup, and I stayed on through the Hornetsecurity acquisition. I know this product from the inside, so weigh what I say about it accordingly. That brings me to my first point. Whatever backup application you choose for your given environment SHOULD be boring and predictable. You should know it so well that it never surprises you.

This goes for the architecture side of the discussion as well. Example: I have an OLD ReadyNAS with 6 disks in a RAID 6. I’ve had this thing forever. I know it, and it just works. I know it so well, in fact, that I’m also deeply aware of its faults. It’s old, therefore it runs an old insecure version of SMB. So when I rearchitected the backup strategy for the lab, I planned for that. The ONLY system able to talk to that SMB share is the system the backup software is running from. Plus, that system has the minimal amount of permissions needed to do the job and said permissions are tied to a dedicated service account specifically for that purpose.

Overkill for a homelab? Maybe. But I’ll die on the hill of “You should treat your lab like production”.

That brings me to my final point. In the age of increasingly expensive hardware, at least make the attempt to make use of aging hardware. Backup is critical yes, but you can architect effective and safe backup storage, even with aging hardware. For example, I used RAID 6 with the understanding that yes, sometimes 2 drives fail simultaneously (and i’ve seen it happen more than once). I also paired the backups with an offsite backup location using an Azure storage account. So the solution is hardened against multiple local drive failures as well as whole NAS failure with the full offsite copies, immutability, and 1 year’s retention.

Backups are for failure situations, but make sure you architect your backup deployment for failure as well.

And now, back to our regularly scheduled programming…. where, as it happens, I end up making a very similar argument about a completely different layer of the stack.

The latest on the SysAdmin Weekly Podcast

Episode: Windows Admin Center in 2026: Good Tool, Broken On-Ramp? (Episode 054) Topic: Four hours, a from-scratch certificate authority, and a stack of error messages pointing the wrong direction, all to reach the login screen of a free Microsoft tool.

Why this one is a good listen:

• Andy and Eric Siron walk the full gauntlet: the 60-day self-signed cert trap, an ERROR_DS_RANGE_CONSTRAINT that blamed the wrong thing, a web server template that quietly refuses computer requests, and a blank SAN that kills the HTTPS binding without telling you.

• The tool underneath is good. The on-ramp is where SysAdmins give up, and giving up on the on-ramp is how a useful tool ends up unused across an entire industry.

• If you are moving to an all Windows Server-Core fleet, this is the install you are about to do. Also covered: Azure Arc and what Microsoft actually wants out of WAC, plus the domain-join debate that refuses to die.

Watch on YouTube

Listen on Spotify

The Take

This may be a hot take, but the vast number of different Hyper-V management options in Windows and the Microsoft Cloud is a strength of the platform and not a detriment (Like I used to think it was).

Many SysAdmins in the industry started their virtualization management journey with VMware vSphere. vSphere is known for its powerful management story in that much (if not all) of the day to day management can be done from one central console. Microsoft’s Hyper-V has long been criticized for its use of multiple disparate management tools. The list, which is extensive, includes:

• Hyper-V Manager

• Failover Cluster Manager

• System Center Virtual Machine Manager (SCVMM)

• PowerShell

• Windows Admin Center

• Management from Azure via Azure Arc

• Windows Admin Center vMode (New and in Public Preview)

Even though I’ve spent many years as a Microsoft MVP in the Cloud and Datacenter management category, I used to be on the “VMware’s management story is better” side of the discussion, even though I was of the opinion that, overall, Hyper-V was the better solution. Over the last few years my stance has changed and I’ve come to appreciate the management options with Hyper-V.

Do I think there could be a better “single-pane-of-glass” management solution for Hyper-V? Sure, and maybe Windows Admin Center Virtualization Mode is it. It’s still early days on that particular tool and system requirements for it are… intense, but time will tell. Outside of WAC vMode though there are a number of seemingly disparate tools that I’ve come to appreciate their individualness. Each tool has it’s own strength and together they present a whole management story that does NOT have a single point of failure. For example:

• If I need to manage a stand alone box - Hyper-V Manager.

• Managing a small cluster? - Failover Cluster Manager

• Large Cluster? - Use SCVMM

• Automation task - Call PowerShell

• Someone needs a simple webUI and hates MMC? - Windows Admin Center (if you can get it installed =/)

• Manage things from the same control plane as Azure Services? - Azure Arc

Again, the choice and the flexibility IS THE POINT. I guess that’s the part that I’ve been stuck on all these years. I was spoiled by the single tool in my VMware days and never stopped to think about the advantages outside of that management model. What happens if that single tool breaks? Sure, I have PowerCLI as a backup on the VMware side, and I can think of maybe 3 times in my career that the vSphere UI was unusable, so unlikely. But, as a SysAdmin who’s been doing this for 24 years, I’ve learned that shit will hit the fan when you least expect it.

This is also not to mention the fact that Broadcom has been throwing around legal threats and generally making vSphere too expensive to operate for some organizations. Do I want a company like that, who uses relicensing and contract scares, to have a potential hold over the limited management avenues into my virtualization ecosystem? Probably not…. At least with Hyper-V I have multiple management frameworks and paths to get the job done, even in the worst of circumstances.

Hot take? Maybe, but that’s where I find myself these days on this particular debate.

What about you? Let me know in the comments!

Community Signal

Kristopher Turner - “PowerShell Automation Patterns (2026 Edition)” - A multi-layer approach to driving Hyper-V without touching a console: config files, modules, declarative DSC v3, and a CI/CD pipeline on top. The useful detail that Kristopher lays out is that, to quote the article: “the Hyper-V module (roughly 245 cmdlets) and FailoverClusters (roughly 90) now run natively under PowerShell 7 with no compatibility shim”. This used to drive admins BONKERS. Turner frames the whole stack as the free, version-controlled answer to VMware Host Profiles and Update Manager, meaning the road into your fleet that survives a broken GUI is the one you scripted. A GREAT read overall!

Tool of the Week

Apache Guacamole - A clientless remote desktop gateway that serves RDP, VNC, and SSH sessions to a plain browser tab, with nothing installed on the machine you are sitting at.

The reason it earns the slot this week is that it is a second road in. When your primary console is broken, relicensed, or waiting on a certificate you cannot mint, a gateway that speaks the native protocols directly does not care. It handles LDAP, SAML, OAuth, and TOTP for auth, and file transfer is drag-and-drop into the session.

Honest scope: this is a real deployment, not an appliance. You are running the web app, the guacd proxy daemon, and a database, and Docker Compose is the sane way to do it. And be clear-eyed about the irony, because a gateway everything routes through is itself a single point of failure. Guacamole is worth running as an additional path to your infrastructure, SHOULD YOU NEED IT (not everyone does). Put it behind a VPN and keep your direct RDP and SSH paths working. Remember though, this could potentially be another attack surface for the given environment, so secure it and treat it as such.

Quick Win of the Week

Pick the console you reach for most, then go do one routine task without it.

Not a hard task. A boring one you did last week. Check a VM’s memory assignment, restart a service on a member server, pull the last ten failed logons. Do it from PowerShell, or the MMC snap-in, or whatever your second road happens to be, and time yourself.

If it takes ten minutes, you have a working fallback. If you find yourself opening the primary console to look up how to do it without the primary console, you just found out how much of your operational capability lives in one UI. That is the number worth knowing before the bad day, not during it.

Fun Retro SysAdmin Fact

The 1988 Berkeley paper that gave us the word RAID, by Patterson, Gibson, and Katz, defined exactly five levels and RAID 6 was not one of them; dual parity came later, and the industry also quietly swapped the “I” from “Inexpensive” to “Independent” once vendors started charging real money for the arrays.

Until Next Week

Go find out which single thing in your stack everything else is quietly leaning on, because it already knows and you don’t.

Stay Frosty,

Andy

SysAdmin Weekly

• Rebuilt damned-near the whole lab on Windows Server 2025: WS2025 Hyper-V host, Debian guests, plus DC, CA, and fileserver VMs (all 2025).

• Homelabs are how you meet the newest version of the thing, and its rough edges, before a change window forces the introduction.

• The Take: “supported” is a claim a vendor makes, not a promise the software keeps; test the mainline path in the lab from time to time, not just the weird edge cases.

• Podcast episode 053: how to survive as a solo SysAdmin, with Eric Siron.

• Tool of the Week is Smallstep step-ca, an open-source internal CA that makes lab certificates a lot less painful than they were this month.

  👋 New here? Subscribe to SysAdmin Weekly for great IT content, and SysAdmin therapy.

From the Console

I spent the last few weeks tearing my lab down to the studs and rebuilding (again) on Windows Server 2025. Hyper-V now runs the virtualization layer, and the part that still makes me grin is what sits on top: a stack of Debian VMs, right alongside a domain controller, a certificate authority, and a file server, all three of those on Server 2025 as well. Windows hosting Linux guests alongside the AD estate. That is the lab for now.

This is the whole reason any of us run homelabs. Not the rack lights. It is the one place you get to meet the newest version of the thing on your own terms, before a customer or a change window makes the introduction for you. Staying current is not a nice-to-have in this job; it is the job.

What do the “what’s new in Server 2025” posts skip? Labbing the new stuff does not just show you the new capabilities. It shows you which of the vendor’s own instructions (and docs) quietly do not hold. I went to stand up Windows Admin Center on that fresh fleet, the documented and supported way, and lost most of an afternoon to it. The supported installer walked right up to the certificate step and silently failed to bind the cert to HTTPS (amongst many other issues), on a clean box, following the official docs to the letter. Four hours, and about ten minutes of that was genuinely my fault. I am writing the full autopsy up as a blog post for AndyOnTech.com, so I will not spoil the ending here. The short version is the thread running through this whole issue: “supported” is a claim a vendor makes, not a promise the software keeps, and the lab is where you find that out on your own time instead of in production.

The latest on the SysAdmin Weekly Podcast

Episode: How to Survive as a Solo SysAdmin: Where to Start When You’re the Only One Topic: When you are the only person between a working business and total collapse, the job stops being about doing everything and starts being about deciding what not to do this week.

Why SysAdmin’s should tune in:

• It is a survival guide for the team of one: triage before projects, documentation as insurance, and buying your time back with automation and the right managed services partner.

• Ignore it and you burn out, or you get hit by a bus and the business finds out the hard way that everything important lived in your head.

• For anyone who got hired into, downsized into, or promoted into owning the whole environment

• Plus a News React on Windows Server hot patching and a Nerd Hour on Debian 13.6 Secure Boot certs.

Watch on YouTube

Listen on Spotify

The Take

Vendors publish a “supported path” for a reason, and we lean on it for a reason. It is meant to be the route that has been tested, documented, and blessed, the one you can trust when you do not have time to reverse-engineer the product. So here is the uncomfortable thing I got reminded of this week. Supported is a claim, not a guarantee. The documented, supported way to do a routine thing with Windows Admin Center quietly failed multiple ways throughout the installation / setup, and nothing in the process warned me or provided meaningful error messages.

When you are the solo or lean-team admin like the theoretical solo SysAdmin we discussed in this week’s featured podcast episode, you plan your maintenance windows around the assumption that the supported path works. You do not budget four hours to install a management tool the vendor says installs in twenty minutes. The exotic edge cases you already distrust, so you test those. It is the boring, supported, “this obviously works” step that sometimes surprisingly wrecks your evening, precisely because you had no reason to give it a second look.

So test the happy path, not just the weird one. The install everyone says just works, the migration the docs call routine, the failover you assume is fine because the vendor says “It’s fine… trust me bro”. Run it once in the lab before you run it live. Not because you are paranoid, but because “supported” and “actually works in your environment” are two different sentences, and the gap between them is your outage. I would rather find that gap in my basement or test/dec environment than in a change window with someone waiting on me.

Community Signal

James van den Berg - “Windows Admin Center 2511 and the Security of Windows Server” - Van den Berg is a long-running Microsoft MVP for Cloud and Datacenter Management, and this is a clean walkthrough of using Windows Admin Center to drive the Server 2025 security surface: OSConfig baselines for CIS and DISA STIG, Secured-core, TPM attestation, and VBS. What earns it the slot this week is that he tells you to test it in a lab before you point it at production. That is the exact reason my own lab exists, and it is a reminder that WAC, on-ramp gripes aside, is a genuinely useful console once it is running.

Florian Burnel / IT-Connect - “Debian 13.6 Is Out: Here Are the Key Changes” - Burnel is a Microsoft MVP for Cloud and Datacenter Management, and his rundown of the Debian 13.6 point release is a near-perfect illustration of this week’s whole point. Microsoft’s old UEFI Secure Boot certificate authority, the one that has quietly signed bootloaders for well over a decade, expired in June 2026, and Debian 13.6 ships the fwupd 2.0.20 update that rotates affected machines onto the Microsoft UEFI CA 2023 certificate before a future bootloader update leaves a Secure Boot box unable to start. Nobody signed up for “my certificate expired, so my server will not boot,” and yet here we are. If you run Debian anywhere in the estate, read this before your next kernel or shim update makes the decision for you; it is the exact terrain Eric and I walk through at Nerd Hour this week.

Tool of the Week

Smallstep step-ca - An open-source, online certificate authority with a built-in private ACME server, so your internal boxes can automate certificate issuance and renewal the same way the public web does with Let’s Encrypt.

If you have ever wanted TLS everywhere in the lab without hand-rolling OpenSSL commands or standing up a full ADCS deployment, this is the lighter path. It speaks ACME, so certbot, Caddy, Traefik, nginx, and cert-manager all just work against it, and it issues short-lived certs with automated renewal so you are not babysitting expirations. Honest scope: it is Apache-licensed and actively maintained, and it is aimed at DevOps and homelab use, not at replacing an AD-integrated enterprise PKI. There is no OCSP or CRL and no Active Directory Certificate Services integration, so if you need GPO autoenrollment and AD-authenticated templates, that is still ADCS territory. One caution: it has patched a run of provisioner authorization CVEs over the past year, so run a current release (0.30.0 or newer) and do not expose the SCEP endpoint if you are not using it. For a lab, or for internal service certs on a private network, it removes a lot of pain.

Quick Win of the Week

Take the last thing that cost you half a day of troubleshooting and write the ten-minute version of it. Not a polished KB article; the terse “here is what was actually wrong and here is the command that fixed it” note that your future self, or the next admin, will be desperate for at 2am. Drop it wherever your hit-by-a-bus documentation lives. The four hours are already spent; the only question is whether anyone, including you, ever gets those hours back.

Fun Retro SysAdmin Fact

Windows Admin Center did not start life with that name. Microsoft unveiled it at Ignite in September 2017 under the codename “Project Honolulu”. I was there and remember it! Microsoft pitched as the browser-based console that would finally pull the scattered MMC snap-ins into one pane of glass, and it ran under that codename all through the preview. It did not pick up the Windows Admin Center name until it went generally available on April 12, 2018. So the tool that ate my afternoon this week is only about eight years into carrying its real name, and apparently still working a few of the install-day kinks out of its system.

Until Next Week

Run the boring supported step in the lab once before you trust it in production. Remember…. supported and working are not the same sentence.

Stay Frosty,

SysAdmin Weekly

• The most valuable skill I saw at a security conference this week had nothing to do with security: it was the ability to say what a technical control does in a sentence a CFO can repeat.

• Highlighting 052 of the podcast: Andy and Eric Siron dig into why homelab and server hardware went vertical in 2026, who is eating the supply, and why the prices probably do not come all the way back.

• The Take: hardware just got expensive in the worst budget climate in years, and “DDR5 is up 125%” is not a sentence that funds a refresh; the business-outcome version is.

• Community Signal: Packet Protector’s episode on translating security work into business outcomes, with concrete moves like reading your company’s own earnings call before you ask for money.

• Tool of the Week is Netdata, because the fastest way to win a hardware argument is to walk in with the utilization data instead of a vibe.

  👋 New here? Subscribe to SysAdmin Weekly for great IT content, and SysAdmin therapy.

From the Console

I spent this week at a security conference in the Dallas area, and the most useful thing I heard all week had nothing to do with security.

It was at XChange Security, a room full of MSPs and MSSPs who run other people’s infrastructure for a living. These are not people who need convincing that email authentication matters. And yet the conversation that kept surfacing, over and over again, was not about attack surface or detection coverage. It was about how to keep a security practice funded when the person signing the checks does not understand a single thing you do.

One thing I’ve learned after being in tech as long as I have…. No executive has EVER cared how your DMARC, DKIM, and SPF records are configured. That is a genuinely interesting conversation to have with me, at nerd hour, over a beer. Like…seriously. I’ll geek out over DMARC with all the other techs. That said, it’s a terrible conversation to have with your CFO. What the CFO wants to know is whether someone can impersonate the company to its customers, its vendors, and its bank. That is the whole question. Same three records, completely different sentence.

We are in a stretch where a lot of IT departments are under a microscope on both budget and headcount. In that climate, the ability to do the technical work is table stakes; it is assumed, and it does not save your seat. The thing that saves your seat is being the person who can stand in front of leadership and translate the work into an outcome they can act on. Not “we deployed the thing.” Instead: “we closed the gap that lets someone pretend to be us.”

Call it the translation layer. It sits between what you actually did and what the business heard, and most technical people never build it. The ones who do are the ones who are still employed when the spreadsheet gets tight.

The latest on the SysAdmin Weekly Podcast

Episode: Why Is Homelab Hardware So Expensive in 2026 (and When Will It Get Cheaper)?

Topic: Andy and co-host Eric Siron get into why memory, NAND, and storage prices went vertical in 2026, who is actually absorbing the supply, and what practitioners should do about a hardware market that stopped making sense.

Why it’s worth your time:

• It puts real numbers on the pain: a Raspberry Pi board that was $120 is now $305, write-hardened NVMe drives are priced into orbit or flat out of stock, and Andy admits to pricing used flash storage on eBay before he pumped the brakes and re-architected his whole lab.

• It explains the mechanics you will be quoting to your own boss soon: the AI buildout is eating memory and NAND capacity, three companies control roughly 95% of DRAM, a new fab takes three to five years, and Gartner has prices settling permanently above pre-shortage levels.

• It lands on the skill this crunch is quietly reviving: right-sizing hardware to the workload instead of reflexively throwing spec at the problem, which is the discipline the cheap-hardware era let atrophy.

Watch on YouTube

Listen on Spotify

The Take

Your next hardware refresh got a lot more expensive, and you are going to have to go ask someone for the money. In this budget climate….. Good luck.

The episode above explains why the prices are what they are, and the short version is bleak: this is structural, not a blip, and it may not fully reverse for years. That is the part you already feel every time you open a vendor quote. Now…what happens next is that you have to walk into a room and justify a bigger number than last year to someone who was already looking for reasons to cut. And the way most of us instinctively make that case is exactly the way that fails.

Here is the failure mode. You lead with the mechanics, because the mechanics are true and you find them compelling. “DDR5 is up 125%, NAND is worse, the drives I need are on three-month lead times, and the refresh I scoped at forty grand is now closer to sixty.” Every word of that is correct. None of it funds anything. The person across the table heard “prices went up and IT wants more money,” which is the least persuasive sentence in the building.

Now translate it. “The storage under our billing system is past warranty and out of spare capacity. If it fails, we cannot invoice customers until it is replaced, and replacement parts are now measured in months, not next-day. I want to refresh it while we can still choose the timing instead of having the timing chosen for us.” Same request. Same dollars. But now it is a business-continuity decision with a clock on it, not a line item that went up. That is a sentence a CFO can carry into their own meeting and repeat.

That is the whole game right now, and it is the same muscle from up top in a different room. The hardware crunch did not just make gear expensive. It made the budget conversation harder at the exact moment the budget conversation got more hostile. The admins who get their refreshes approved this year will not be the ones with the best spec sheets. They will be the ones who showed up with the utilization data and the business consequence, and left the part numbers in their back pocket unless someone asked.

Community Signal

Packet Protector - “Translating Security Objectives into Business Outcomes” - Hosts JJ Minella and Drew Conry-Murray sit down with Eyvonne Sharp, who spent twenty-plus years going from technical IC to leadership, and the whole thing is a practical manual for the translation layer I keep going on about. The moves are concrete and a little uncomfortable in a good way: read your own company’s earnings call or 10-K to find out what leadership actually cares about before you ask for a dime, quantify your work in revenue or mission terms instead of hours saved, and drop the fear-based “reputational damage” pitch because seasoned executives have heard it a hundred times and tuned it out. If The Take this week resonated, this is the long-form version from people who have sat on both sides of the table.

Tool of the Week

Netdata - a free, open source (GPLv3+) monitoring agent that gives you per-second CPU, memory, disk, and network metrics on every node, with real-time dashboards and zero configuration to get started.

I am putting it here for a specific reason tied to The Take: the fastest way to win a hardware argument is to stop arguing and show the graph. Netdata is how you find out that the database server everyone swears is “maxed out” is sitting at 11% memory utilization, or that the box you were about to leave alone is genuinely out of headroom. That is the difference between right-sizing on data and right-sizing on a hunch, and it is the receipt you bring to the budget conversation. Honest scope: the agent is free and open source and runs comfortably on a homelab node at around 5% of a core and 150MB of RAM, and it scales up to real infrastructure. The catch is retention and the single-pane view across many machines. Long local history and a unified dashboard across a fleet lean on Netdata Cloud, which has a usable free tier for individuals and paid tiers once you are a team, or on self-hosting parent nodes if you want to keep everything in-house. For a homelab or a small shop that just needs to see what is actually happening right now, the free agent alone earns its keep.

Quick Win of the Week

Take the last status update or upgrade request you sent up the chain, the real one, and rewrite the opening sentence so a non-technical executive could repeat it out loud without you in the room. Cut every acronym and every part number from that first line. If you cannot state the business outcome or the business risk in one sentence, that is not a writing problem, it is a sign you have not figured out why the work matters to anyone holding a budget, and better to learn that at your desk than in the meeting. Fifteen minutes, one sentence, and you will use the skill again within the month.

Fun Retro SysAdmin Fact

Hardware sticker shock is not new: in 1956, IBM’s 305 RAMAC, the first computer to ship with a moving-head hard disk, stored a whopping 5 million characters (about 5MB) across fifty 24-inch platters and leased for roughly $3,200 a month, which is north of $36,000 in today’s money, so the next time a NAND quote makes you wince, remember your homelab holds a few million times that for the price of a nice dinner.

Until Next Week

The technical work is the part you already know how to do; the sentence you say to the person holding the budget is the part that keeps you employed.

Stay Frosty,

Andy

SysAdmin Weekly

• Thunderbird 145 now speaks native Exchange (EWS with modern OAuth2), which means an open source mail client that finally works with Microsoft 365 without legacy auth or IMAP duct tape.

• New episode 051: Andy and returning guest Clay Tamam tour what actually runs in their labs, hardware to hypervisors to the Graveyard of gear they powered off.

• The Take: “end of life” is often a status you assigned, not a permanent fact; the tools you wrote off deserve a revisit cadence, because the reason you quit them sometimes gets fixed.

• Community Signal: Brandon Lee on the popular homelab advice he no longer follows, which is the same right-size-the-lab discipline the episode is built on.

• Tool of the Week is Jellyfin, the open source answer to the latest round of Plex price and paywall changes.

From the Console

I did not expect to feel anything about an email client in 2026. Then Thunderbird went and surprised me.

For years, Thunderbird was effectively dead to me for anything touching Microsoft 365. If you wanted it to talk to Exchange Online, you were bolting on a paid add-on or falling back to IMAP and legacy authentication, which is to say the exact protocols Microsoft has spent years trying to kill off, and for good security reasons. So on my Debian box, the machine I actually enjoy working on, I had no real open source mail client for my M365 mail. I lived in a browser tab and told myself that was fine.

Version 145 changed that. Thunderbird now speaks Exchange Web Services natively, uses the standard Microsoft OAuth2 sign-in, and detects the account settings for you. No add-on. No legacy auth. No pretending IMAP is a plan. I set it up, it just worked… mostly (after I accounted for bug 1847846 due to my large mailbox size). After that, I sat there mildly stunned that a piece of software I had mentally filed under “lost cause” had quietly clawed its way back into my daily driver rotation.

We wrote a whole issue about this once. Issue #22 was about open source as a lifeline: the exit door you keep unlocked for when a vendor changes the terms on you. This is the other half of that same coin. Sometimes the lifeline is not a new tool at all; it is one you already knew, left for dead, that got picked back up and reworked to fit the modern world. More on that below.

The latest on the SysAdmin Weekly Podcast

Episode: What I Actually Run in My Homelab (and Why)

Topic: Andy is joined by returning guest Clay Tamam, a working SysAdmin in the Netherlands, for an honest tour of what is really sitting in their labs: the hardware, the hypervisors, the services they use every day, and the reasoning behind each choice.

Why it’s worth your time:

• It is the antidote to homelab-flex culture: two practitioners talking about what earns its place and what got powered off, not what looks impressive in a rack photo.

• Andy walks through tearing a four-node Kubernetes cluster down to five VMs on a single Debian box, and Clay builds a rack from scratch on a real budget, so you get both ends of the spectrum with the tradeoffs spelled out.

• It closes with the Graveyard: the gear and services that got shut off and why pruning is part of the discipline, not a failure of it.

Watch on YouTube

Listen on Spotify

The Take

“End of life” is a status you assign as often as one a vendor declares. And the two are not the same thing.

The podcast episode (above) closes on the Graveyard, the pile of gear and services you power off because they stopped earning their place. That discipline is correct. Most labs, and most production stacks, are carrying dead weight that nobody has had the nerve to switch off. Don’t get me wrong, removing technical debt is a good thing (in most cases) but the graveyard is not a one-way door. Sometimes a tool you buried was not actually dead. It was just dead to you, because the specific thing that made it unusable had not been fixed yet.

Thunderbird is my case in point this week. I wrote it off for M365 work years ago, for a completely valid reason at the time, and then I stopped tracking it. That is the trap. “I wrote it off” quietly turns into “I stopped paying attention,” and the two feel identical right up until the day the project ships the exact fix you gave up waiting for. Thunderbird 145 added native Exchange support over EWS with modern authentication, and the coverage was blunt about how long people had wanted it. The reason I quit it got fixed. I just was not looking.

Let’s be specific about the discipline, though, because this is not nostalgia. A tool earns its way back on current merits, not sentiment. Is it actively maintained? Does it meet your real requirements today? What is the catch? Thunderbird bet on EWS, a protocol Microsoft is slowly phasing out in favor of Graph, so the honest read is that email works now, calendar and contacts are still in progress, and the team is already building toward Graph for when EWS finally sunsets. That is a tool worth adopting with eyes open, not a fairy tale. The point is the cadence: put the tools you wrote off on a revisit schedule instead of treating a one-time verdict as a life sentence. The graveyard occasionally coughs one back up, and you want to be the admin who noticed.

Community Signal

Brandon Lee - “I No Longer Follow This Popular Home Lab Advice” - Brandon has decades in the field, and this is the same right-size-your-lab thesis Episode 051 of the podcast (with a mention in 052!) is built on. He makes the case for using Kubernetes only when orchestration actually solves a problem, applying high availability selectively instead of everywhere, and treating a couple of services (email and password managers among them) as reasonable exceptions where managed reliability beats self-hosting. It is a rare homelab piece that argues for running less, and it pairs directly with my Kubernetes-to-VMs teardown in episode 051.

Tool of the Week

Jellyfin - a fully open source, self-hosted media server for your movies, shows, and music, with no accounts to create, no cloud dependency, and no feature you rely on sitting one pricing update away from a paywall.

The reason it earns the slot this week: Plex just tripled the price of a new lifetime Plex Pass to $749.99, effective July 1, on top of already gating remote streaming behind a paid pass, which is exactly the vendor-changes-the-terms moment that sends people looking for the exit. Jellyfin is that exit, and it is genuinely free with no premium tier gating the core experience. Honest scope, though. This is not a turnkey Plex clone. Hardware transcoding takes real configuration to get right, the client apps across smart TVs and streaming boxes are less polished than Plex’s, and Jellyfin ships no built-in remote-access magic, so getting to your library from outside the house means you bring your own reverse proxy or a mesh VPN like Tailscale (which came up in the episode for exactly this reason). For a homelab that already runs a hypervisor and a bit of networking, that is a comfortable weekend project. For a non-technical household that just wants to press play, Plex is still the smoother ride, and it is fair to say so.

Quick Win of the Week

Pick one tool you wrote off more than two years ago, one you filed under “dead” and stopped tracking, and spend thirty minutes with its current changelog and release notes. Not to adopt it. Just to find out whether the specific thing that made you quit it has been fixed. Thunderbird and Microsoft 365 was mine this week; you almost certainly have your own candidate sitting in your mental graveyard. The goal is to make “I wrote it off” a decision you revisit on a schedule, not a verdict you handed down once and never appealed.

Fun Retro SysAdmin Fact

Thunderbird 1.0 shipped on December 7, 2004, which makes Mozilla’s mail client older than most of the “modern” SaaS platforms built to replace it; it was very nearly left to rot when Mozilla stepped back from active development in 2012, and here it is twenty-one years later adding native Exchange support and quietly having a renaissance.

From My Other Corner of the Internet

AndyOnTech

The reconstruct-project-history piece I had been sitting on is finally live: Can Claude Code Reconstruct a Project History You Never Documented?. It walks through pointing Claude Code at nothing but a repo’s git history and having it rebuild the changelog and decision trail that no human ever sat down to write. If you have a months-old project you no longer fully understand and you’re asking yourself the question “how did I get here?”, this is the practical, boring-value use case for the tooling, not the hype one.

Until Next Week

Prune what stopped earning its place, but keep a list of the tools you buried; a few of them are going to knock.

Stay Frosty,

Andy

SysAdmin Weekly

• Right tool for the right job is a discipline, not a budget line: stop sizing your tech to your ego and start sizing it to the task.

• New episode 050: how to run a blameless incident postmortem that ends with a root cause instead of a name, whether you’re a war room of sixty or a shop of one.

• The Take: your own postmortem is half the skill; reading everyone else’s, including your vendors’, is the other half, and their transparency is a procurement signal.

• Tool of the Week is OneUptime, an open-source stack that captures the incident timeline automatically so your postmortem has real material to work from.

• Quick Win: steal the show’s free postmortem template and fill in the boilerplate now, before the pager goes off.

From the Console

One of the most expensive habits in tech has LONG been reaching for the biggest thing on the shelf just because it happens to be sitting there.

I caught myself doing it last week. I had a two-paragraph text file that needed summarizing, and my hand instinctively went straight for the largest, most capable AI model I had access to (because i’ve been doing a lot of complex work lately). Had I not caught myself, that’d be like renting a semi to move a bag of groceries. Opus does not need to summarize your grep output. Haiku is not going to architect your app. The skill is not knowing which model is “best,” it’s knowing which one fits the job in front of you, and most jobs are small.

The lab version of this is worse, because the lab version has blinking lights. Nobody needs a half-rack SAN to hold a few text files and a 2GB media library, and yet the homelab channels are full of gear that exists mostly because it was available and the owner could. I have been guilty of this too. Right tool for the right job is not a spending rule, it’s a discipline and arguably MORE important in today’s world of insane hardware prices that it used to be. The moment you size the tech to the task instead of to your ego, everything gets cheaper, quieter, and easier to reason about.

The latest on the SysAdmin Weekly Podcast

Episode: How Do You Run a Blameless Incident Postmortem?

Topic: Andy and Eric Siron pull from combined decades of incident reviews to break down what a postmortem actually is, what kind of outage earns one, who belongs in the room, and why the follow-through is the whole point.

Why this one matters:

• An effective postmortem turns an outage into tangible fixes instead of finger-pointing, which is the difference between an incident you learn from and one you repeat.

• Skip it and the same failure comes back, because nobody wrote down the contributing factors and nobody owned the action items.

• It’s for anyone who owns uptime (which is MOST SysAdmins), and the discipline scales: sixty people in a war room or just you writing a summary for one nervous boss.

Watch on YouTube

Listen on Spotify

The Take

Running your own postmortem is half the skill. Reading everyone else’s is the other half, and almost nobody treats it that way.

Every vendor you depend on has a track record of how they behave when they break. Some of them tell you. In February, Clerk put out a postmortem that named the actual root cause: a Postgres auto-analyze flipped a query plan because the planner’s sample size was too low, the bad plan strangled the database, and roughly ninety minutes later a manual re-analyze brought it back. That is a real root cause, with the communication failures owned in the same document. You read it and you know exactly what happened and what they changed.

Now, that willingness to publish a specific cause is itself a signal, and it correlates almost perfectly with how a vendor treats you at 2am. When GitHub’s own engineering leadership publicly attributes a run of outages to tightly coupled architecture that let local problems cascade, and to systems that couldn’t shed load from misbehaving clients, that is a company telling you how it actually failed. Compare that to the outfits whose entire public record is “we experienced elevated error rates” followed by silence. One of those is a partner. The other is a liability you haven’t been billed for yet.

So make it part of procurement. Before you take a hard dependency, go read the vendor’s status history and their last two or three postmortems. Not the marketing uptime number, the writeups. If they don’t publish any, or if every one is corporate mush with no named cause, you already know how the next outage conversation goes. The postmortem discipline Episode 050 is about does not stop at your own perimeter.

Community Signal

Awesome community work worth your attention.

Lorin Hochstein - “Quick thoughts on GitHub CTO’s post on availability” - Hochstein is a resilience-engineering practitioner (ex-Netflix SRE) who writes some of the clearest incident analysis on the internet, and here he picks apart GitHub’s own writeup on its recent outages. It’s the perfect companion to this issue’s Take: this is what it looks like when a working engineer reads someone else’s postmortem and pulls the real lessons out of it.

Google SRE - “Postmortem Culture: Learning from Failure” - Chapter 15 of the free SRE book, and still the clearest articulation of blameless-not-unaccountable in print. If Episode 050 had you nodding, this is the reference to hand the skeptic on your team who thinks “blameless” means “nobody is responsible.”

Tool of the Week

OneUptime - an open-source, self-hostable reliability stack that rolls monitoring, on-call scheduling, status pages, and blameless postmortems into one platform.

The reason it earns the slot this week: it captures the incident timeline automatically while the fire is still burning, which is exactly the raw material a good postmortem needs and exactly what nobody remembers to collect by hand at 2am. Honest scope, though. This is a full platform, not a single container you set and forget. Self-hosting it is real infrastructure with a database and the works, so for a homelab it’s a weekend project, not a five-minute one; if you just want uptime checks, Uptime Kuma is the lighter tool. And to be straight with you: this is a smaller, less grassroots-proven community than a darling like Uptime Kuma, and it’s an all-in-one bet rather than a best-of-breed one. What earns it the slot is that it’s genuinely fully open on GitHub (Apache 2.0, not open-core), actively maintained, and its postmortem tooling actually exists instead of being bolted on.

Quick Win of the Week

Drop a real postmortem template into your team wiki or repo today, before you need it. We linked a markdown postmortem template that i’ve used over the years in the resources section of the last podcast episode. That version is free, CC BY 4.0, and version-tracked in the SysAdmin Weekly community repo here.

Pre-fill the parts that never change (your severity levels, escalation contacts, and the timeline table headers) so that when something does break, you’re filling in facts instead of formatting a document while the help-desk phone screams.

Fun Retro SysAdmin Fact

On January 15, 1990, AT&T’s long-distance network collapsed for about nine hours and blocked roughly half of all calls placed through it, an estimated $60 million in lost connections, and the root cause was a single misplaced break statement in the C recovery code running on its 4ESS switches: proof, 36 years early, that “blameless” exists because the cause is almost never a person.

Question I Got Asked (and the Real Answer)

The question: “I’m a one-person shop. Do postmortems even apply to me?”

The common (wrong) answer: “No, that’s enterprise process overhead. You already know what broke.”

The real answer: The document scales down to a paragraph; the discipline does not. You will not remember, three months from now, the exact order you tried things at 2am or why the obvious fix didn’t work. A five-minute writeup with a timeline, a root cause, and two action items you actually close is worth more to a solo admin than a forty-page template is to a team that files it and forgets it. Blameless is easy when the only name in the room is yours. The hard part, same as everywhere, is the follow-through.

Until Next Week

Size the tool to the task, write down what broke, and close the action item you keep meaning to close.

Stay Frosty,

Andy

SysAdmin Weekly

• After a few issues of me telling you not to trust the machine, here is the other half of the ledger: the tangible, boring AI wins that are actually worth your time.

• Episode 049 is out: how attackers run local LLMs on their own hardware to generate spear phishing at scale, why “spot the typo” training is dead, and where the real defensive line sits, with Eric Siron.

• The Take: the same accessible AI that attackers point at scaling the bad stuff can be pointed at the valuable, well-scoped work you never had hours for, including understanding the threat landscape itself.

• I rebuilt a four-month-old project’s entire changelog from nothing but the git diffs in about twenty minutes. The scoping that made it cheap is the same scoping that made it good.

• Tool of the Week: Fabric, reusable AI “patterns” you run from the command line against hosted or local models.

From the Console

I have spent the last several issues being the guy telling you not to trust the machine. Doom Discounted, The Off-Switch You Don’t Own, the whole local-inference drum. I stand by every word of it. But that is one half of the ledger, and this week I want to read you the other half.

Because here is the thing I do not say enough when I am busy being skeptical: lately the boring AI wins have been quietly piling up, and they are real. The one that got me this week…. I pointed Claude Code at a four-month-old project of mine that had grown from a single throwaway script into a full workflow, and asked it to reconstruct the changelog I never bothered to keep, from nothing but the git diffs. About twenty minutes later I had an accurate, dated history of a project I could no longer fully trace in my own head. Not magic. Just a tedious, genuinely useful job that was never worth a human afternoon, suddenly worth twenty minutes.

That is the whole theme this week. Not “AI is amazing.” The lesson this week is that the floor on boring-but-valuable work just dropped, and we should go use that…..

And now, back to our regularly scheduled programming.

The latest on the SysAdmin Weekly Podcast

Episode: 049 - How Do Attackers Use Local LLMs to Phish at Scale?

Topic: Andy brings his InfoSecurity Europe session to the show, and Eric Siron joins to walk through how threat actors run open-weight models on their own hardware to generate targeted spear phishing at scale, in any language, with no internet and no guardrails, and where the real defensive line actually sits.

Why this one matters:

• The guardrail gap, made concrete: ask a hosted model to write a phishing lure and it refuses; pull the right open-weight model onto a laptop and that refusal layer simply is not there. The demo is several tailored spear-phishing lures in ninety seconds.

• Why “spot the typo” awareness training is dead, and why verification culture plus strong email authentication (SPF, DKIM, DMARC) is what actually carries the load now

• Plus the nerd hour and news react: Washington pumping the brakes on Fable, the New York ghost-gun-printing law, and the Google AI liability question

Listen on Spotify

Watch on YouTube

The Take

This week’s episode (above) is about attackers pointing cheap, ungoverned AI at the one thing they care about: scaling the bad stuff. Fifteen tailored spear-phishing lures in ninety seconds, no guardrails, no internet connection required. That is the threat, and it is real. Remember though, that the exact same accessibility cuts the other way, and most of us are barely using it.

Earlier this month I built a small tool that reads the places SysAdmins actually congregate, the subreddits, the forums, the threads nobody has time to keep up with, and correlates what people keep struggling with. Not vibes. The recurring, in-their-own-words pain that gets buried under a thousand individual posts. I built it to find things worth covering on the show. It turned into something more useful than that: a standing read on what is actually breaking in the field, and an early-warning lens on how the ground is shifting under us.

That second part matters more after recording Episode 049. The same engine that surfaces “everyone is fighting the same Intune policy bug this month” will also surface “attackers are openly comparing which local model writes the cleanest lures.” Pointing AI at understanding the field, including how the bad actors are using it against you, is not a gimmick. It is one of the highest-value things a defensive team can do with this technology right now, and A LOT of orgs aren’t doing it.

Here is the part that ties the changelog and the community tool together, and it is the whole game: both worked because I scoped them hard. Tell the model exactly what to look at, exactly what to ignore, and exactly what good output looks like, and it earns its keep. Hand it a vague “go figure out my industry” and you get expensive mush. The tangible AI wins are not the demos. They are the narrow, well-scoped, deeply boring jobs you never had the hours for. If you want the longer version of that argument grounded in day-to-day IT work, I wrote it up here.

Community Signal

Simon Willison - “Large Language Models can run tools in your terminal with LLM 0.26” - If The Take is the argument that well-scoped AI is the real win, this is the reference implementation of that idea from a credible independent voice in the space. Willison’s llm is the opposite of the chat-window-you-babysit: a command-line tool that pipes input in, logs every prompt to a queryable SQLite database, and now hands models controlled access to tools you define as plain functions, against hosted models or local ones through Ollama. The post is about a year old now, but it is the cleanest articulation of the Unix-philosophy approach to AI that the GUI hype keeps drowning out, and it pairs directly with Fabric below.

Tool of the Week

Fabric - an open-source framework that turns AI into a set of reusable command-line “patterns” you pipe text through, instead of a chat window you have to babysit.

Fabric is the tool that makes the “scope it hard” lesson above repeatable. Each pattern is a saved, version-controlled prompt for exactly one job: summarize this, extract the action items, analyze this log, and you run it from the terminal with Unix pipes like any other command. It talks to hosted models (Claude, GPT, Gemini) and to local ones through Ollama, so sensitive input can stay on hardware you own. Honest scope: it is genuinely useful for a solo practitioner or a small shop today, but it is not an enterprise platform with RBAC and audit trails, so treat it as power tooling for you, not a sanctioned production service for the org. MIT licensed, north of 40,000 stars, and actively developed as of this month.

Quick Win of the Week

Pick one repository you actually rely on that has no changelog, or one where you started the changelog too late, and have an AI backfill its history from the git diffs. The trick is entirely in the scope. Hand it the short list of files that define how the thing works, the modules, the pipeline definitions, the runbooks, not every variable tweak, and tell it to read only the diffs touching those paths. Tell it to leave any existing entries untouched, and give it the exact output format you want. Then read the result against the diffs before you trust it, because the history is the authority and the summary is not. Twenty minutes, and a project nobody on the team could explain end to end suddenly has a paper trail.

Fun Retro SysAdmin Fact

The AI that fools people is not new; it is sixty years old. In 1966, MIT’s Joseph Weizenbaum wrote ELIZA, a simple pattern-matching script that imitated a therapist by turning your own statements back into questions, and it was so convincing that his secretary, who knew exactly how trivial the program was, reportedly asked him to leave the room so she could talk to it in private. Weizenbaum spent much of the rest of his career unsettled by how readily people projected real understanding onto a machine that had none, a reaction we now call the ELIZA effect.

Source: Smithsonian Magazine - “Why Joseph Weizenbaum, Creator of the World’s First Chatbot, Turned Against AI”

Until Next Week

The skeptic in me is not going anywhere, but it’s important to state that the boring wins are real, and the only way you will believe that is to point the machine at one tedious, valuable job this week and watch it work.

Stay Frosty,

Andy

SysAdmin Weekly

· A frontier AI model can be switched off worldwide in 72 hours, and not by you. Anthropic’s Fable 5 and Mythos 5 proved it this month.

· Clearly it’s not a Claude problem. It is a dependency problem, and it is the strongest case yet for owning the hardware your inference runs on.

· SysAdmin Weekly Podcast Episode 048 is out: the AI doom narrative held up against the actual data with Eric Siron.

· The WEF Future of Jobs numbers say net +78 million jobs by 2030, and the fastest-growing skill categories behind AI and big data are networks and cybersecurity. The work that keeps infrastructure running is on the growth side.

· Tool of the Week: Ollama, the second time around. Same recommendation as Issue 24, sharper reason to act on it now.

From the Console

I stripped my lab down a few weeks ago. Pulled the always-on gear I was not really using, consolidated a couple of roles onto hardware I trust to keep running, and rebuilt the rest with longevity in mind instead of raw horsepower. I expected to give something up for that.

Turns out…..I have not, in fact, given up something for those changes…. Everything I actually depend on is running as well as it ever did, and my power bill came back noticeably lighter. Turns out a lot of what I was running was running because it was already on, not because I needed it on.

Given the recent news about Anthropic… it makes me thing about something. Every box in that pared-down lab is one I control. It does not get a worse model pushed to it overnight. It does not get switched off because of a directive passed down from three companies upstream of me (or the US Government for that matter…). Those cases sounded like a philosophical point a month ago. This week it stopped being philosophical.

And now, back to our regularly scheduled programming.

The latest on the SysAdmin Weekly Podcast

Episode: 048 - The AI Doom Narrative vs the Data: Layoffs, Energy, and Jobs in 2026 Topic: Andy and Eric Siron take the AI fear stories one at a time, the layoffs, the data center energy and water panic, the “office workers are gone in eighteen months” predictions, and put each one next to what the actual data says.

Why this one matters:

· The WEF Future of Jobs numbers, the post-COVID overhiring correction, and the hyperscaler nuclear and cooling commitments, read against the headlines instead of in place of them

· What AI actually looks like at a SysAdmin’s keyboard versus what the press would have you believe it looks like

· Plus the nerd hour: Linus Torvalds on AI-generated bug reports drowning the kernel security list, Edge caught storing passwords in clear text, and a Forgejo and Restic lab cleanup

Listen on Spotify

Watch on YouTube

The Take

A model you build your work around can be turned off in three days, worldwide, and you will not be the one who decides it.

That is not a hypothetical. Anthropic released Claude Fable 5 to the public on June 9, with the wider Mythos 5 class going to a narrow set of partners the same week. By the evening of June 12, both were dark. Not deprecated, not rate-limited; switched off for every customer on the planet. The trigger was a US export-control directive barring access by any foreign national, and because nationality cannot be filtered on a live API in real time, the only way to comply was to pull the models for everyone. Anthropic’s own foreign-born staff included.

The part relevant for us mere-mortal SysAdmins? None of that had anything to do with how we were using the model. You could have had a clean, paid, well-behaved workflow built on it, and it would still have gone dark on Friday night because of a conversation between an executive and a cabinet secretary that you were not in.

And that conversation is its own story. The reporting says Amazon CEO Andy Jassy is the one who flagged the concern to Treasury Secretary Scott Bessent, citing Amazon researchers who got Fable 5 to cough up cyberattack-useful output. Amazon is Anthropic’s investor. Amazon is Anthropic’s cloud host. And Amazon, this week, is the party that walked a competitor’s three-day-old model release into a regulator. I am not going to tell you what to read into that. I am going to tell you those three roles sitting in one company is exactly the kind of thing you want to know about when it comes to the supply chain under your tooling.

To be fair to Anthropic, they pushed back on the severity, noting the same capabilities are already reachable through other public models, and Opus 4.8 and Sonnet kept running the whole time. So this is not me telling you Claude is the problem. Claude is not the problem. The dependency is the problem.

We have been building this case in different shapes for a while. Issue 21 was about compute consolidating into a handful of hands. Issue 25 was about the hardware squeeze landing on your doorstep. Both of those were about cost and supply. This adds a third axis, and it is the sharpest one: availability you do not govern. A model running on hardware you own cannot be revoked by a directive you never saw. That is the whole pitch for local inference, and it just got handed the cleanest example it has ever had. The full timeline is here if you want to read it cold.

None of this means rip the hosted models out of your stack. They are still better than what most of us can run at home, and that gap is real. It means know which of your workflows would simply stop if a model went dark on a Friday, and have an answer that does not require anyone’s permission to switch on.

Question I Got Asked (and the Real Answer)

The question: “Be honest, is AI going to delete my job?”

The common answer: Some executive’s eighteen-month countdown, repeated back with the serial numbers filed off. Half the room nodding, the other half quietly updating their resume.

The real answer: The most-cited primary source on this is the WEF Future of Jobs Report, and the people repeating the doom number rarely seem to have read it. The report projects 170 million jobs created and 92 million displaced by 2030, for a net gain of roughly 78 million. That is real churn, about 22% of the jobs studied, and it is going to HURT in specific places; clerical and administrative roles, cashiers, and data entry are on the wrong side of it.

The more useful number in that same report is the skills outlook: the fastest-growing skill categories through 2030 are AI and big data, networks and cybersecurity, and technological literacy. Those are not the skills of a job being deleted. Those are the skills of the work that keeps the rest of it running.

The displacement is real and worth taking seriously. The version where it deletes the people who keep systems running is not what the data says. Again, Eric Siron and I spend a full episode of the podcast (above) on this distinction, and it is worth the listen, because the gap between the headline and the report is wide enough to drive a career decision through.

Community Signal

James Kilby - “Self-Hosting an AI Stack Using vSphere, Docker and NVIDIA GPU” - The practical version of everything above. Kilby walks through standing up a containerized inference stack on a single NVIDIA Tesla P4 inside vSphere, with the install commands and, the part I appreciated, the real power numbers: roughly 7 watts idle, 50 to 60 under active inference. If you read From the Console and wondered what “own your inference” actually costs to run, this is a concrete answer from someone who measured it. Updated June 2026.

Tool of the Week

Ollama - Open-source local LLM runtime. Pull a model, run it on hardware you own, expose an OpenAI-compatible endpoint any existing tool can talk to.

If you read Issue 24 you have already heard this pitch. The reason it earns the slot a second time is that the argument for it just got sharper. For issues 24, the case was privacy, cost, and not being told you cannot use AI at all. The Fable 5 takedown adds the axis I did not have a clean example for in May: availability you govern. Again…. a model running on hardware you own cannot be pulled by a directive you never saw.

Honest scope has not changed since Issue 24. Local model quality at consumer hardware tiers still trails the frontier hosted models, and for greenfield engineering work you will still reach for the hosted option more often than not. For the workflows that actually need to keep running no matter what happens upstream, this is the fallback The Take is asking you to have. The Quick Win below is “stand it up and prove it works.” An hour of your time, a permission slip from nobody.

Quick Win of the Week

Make a one-page list of every tool and workflow in your shop that calls a hosted AI model, and mark which ones would simply stop if that model went dark on a Friday night. For the one that matters most, pull an open-weight model in Ollama, point the tool at the local OpenAI-compatible endpoint, and confirm it actually runs. You are not switching off the cloud. You are proving you have a fallback that needs nobody’s permission to turn on, before the week you actually need it.

Fun Retro SysAdmin Fact

This is not the first time Washington export-controlled software. Through the 1990s, strong encryption was classified as a munition under the same export rules as weapons, and PGP author Phil Zimmermann spent roughly three years under federal criminal investigation for “exporting munitions without a license” after his code spread overseas; the case was dropped in 1996, and the idea that publishing software could be an arms-export crime has felt absurd ever since. Felt.

Until Next Week

Know which of your systems would keep running if the company upstream of you had a bad Friday.

Stay Frosty,

Andy

SysAdmin Weekly

• The compute consolidation I flagged back in Issue 21 isn’t a forecast anymore; it’s sitting in your parts bin.

• DRAM contract prices jumped roughly 90% in a single quarter, consumer SSDs climbed 147%, and a 16GB Raspberry Pi 5 that launched at $120 now runs $305.

• The hyperscalers didn’t just rent you the cloud; they bought up the fab capacity that makes the parts you’d use to avoid it.

• We pulled an older episode of the podcast to highlight this week > Episode 023 with Eric Siron from the vault: building a real lab without selling a kidney, more relevant now than the day we recorded it.

• This week’s moves: pool mismatched disks with SnapRAID, and actually measure how much write life your NVMe has left.

From the Console

I went looking for a couple of high-endurance NVMe drives last week. Not the bargain-bin kind; the write-tolerant drives you put under a workload that actually does sustained writes. I didn’t find them at a good price. I didn’t find them at a bad price either. They simply weren’t there.

Same story when I priced out more RAM for an existing box. Same story again when I looked at what a Raspberry Pi costs now versus a year ago. Somewhere in the last few months the math quietly flipped: buying new capacity stopped being a line item and turned into a project with a budget meeting attached.

So I’ve stopped trying to buy my way forward. I’m re-architecting what I already own, with longevity as the goal instead of raw horsepower. Make the gear I have last, instead of betting on a parts market that clearly isn’t betting on me.

This is a rant you’re going to be hearing about on SysAdmin Weekly for….. a while… I apologize in advance. =D

And now, back to our regularly scheduled programming…..

From the SysAdmin Weekly Vault

Instead of reaching for the latest episode of the SysAdmin Weekly podcast, we’re reaching for an older episode that fits with this week’s theme around PC hardware and homelabs.

Episode: SysAdmin Weekly 023 - Budget Home Lab Setup for IT Pros (with Eric Siron) Topic: Building a genuinely useful home lab without torching your budget, across five segments: hardware, software, virtualization, networking, and cloud.

Why this one still holds up, and matters more now:

• The hardware advice was always practical: don’t underestimate an old desktop or laptop with enough RAM, and lean on mini PCs as the modern budget chassis. For example, Eric stood up his entire lab on four 32GB nodes for around $4,000 (When we recorded that episode) The underlying logic used? > “cutting edge, not bleeding edge” (skip DDR5, stick with DDR4), reads less like budget tinkering and more like a 2026 survival guide today.

• Eric’s rule of thumb, brick a $100 box, not a $20,000 one, was about learning safely on cheap hardware. In a market where you can no longer reliably buy the $100 box either, it lands as an argument for defending what you already own and being patient with the gear that still works.

• The virtualization tour is still the cheapest path off the VMware tax: Hyper-V on Windows Server’s 180-day evaluation (re-arm it three times for roughly two years of runway), Proxmox if you miss the vSphere feel, libvirt and KVM for the Linux native route. Containers via Docker, Podman, or a small Kubernetes cluster cover the workloads where full VMs are overkill.

• Bonus reality check we did not plan for: Eric flagged in the recording that VMware trial licenses are now gated behind enterprise customer designation. If you ever wondered why I started talking about “the VMware tax” as a thing, that gate is where it started.

The Take

Back in Issue 21, I ran a thought experiment and landed on an uncomfortable answer: the consolidation of compute into a handful of hyperscalers would, eventually, take choice and independent operation away from the rest of us. I framed it as a question and said I thought the answer was yes. I did not expect the proof to arrive this fast, and I did not expect it to come through the supply chain instead of through policy.

So the ugly truth….. The three companies that make the world’s memory have pivoted their fab capacity toward high-bandwidth memory and enterprise parts for AI buildouts, because that’s where the margin is…. it’s simple capitalism. AI is on track to consume roughly a fifth of all DRAM production this year, per TrendForce. The downstream effect isn’t subtle: DRAM contract prices jumped around 90% in a single quarter, consumer SSDs climbed 147%, a 30TB enterprise SSD went from about $3,000 to $17,500, and that 16GB Raspberry Pi 5 went from $120 at launch to $305.

That’s the consolidation thesis, except it isn’t an abstraction about cloud strategy anymore. It’s the reason you can’t buy a stick of RAM or a decent NVMe drive at a price that makes sense. The same demand that’s building hyperscale AI capacity is the demand that’s pricing you out of building anything for yourself.

And that’s the quiet move now……. When the entities building the cloud also control who gets the silicon to build off the cloud, “just rent compute from us” stops being a sales pitch and becomes the path of least resistance, because every other path got more expensive on purpose. I’m not telling you to panic. I’m telling you to notice, and to make the gear you already own last, because owning your own compute is starting to look less like a preference and more like a position you have to defend.

Community Signal

One item this week, and fits right in with these week’s discussion.

Jeff Geerling - “DRAM pricing is killing the hobbyist SBC market” - Geerling lays out how LPDDR memory now makes up the majority of a single-board computer’s cost, which is why anything past 4GB of RAM has drifted out of hobbyist reach. His read lines up with mine: the theme of 2026 is repurposing what you already have, not buying new. If you want the hardware-level receipts behind everything I argued above, this is the post to read.

Tool of the Week

SnapRAID - parity-based protection for an array of ordinary, mismatched disks, which is exactly what you end up with when you can’t buy a matched set of new drives.

Full disclosure before I describe it: I haven’t run this one in production myself yet. It’s on the shortlist for the re-architecting work I’m doing now, and I’m flagging it here because it’s the community-tested answer (Perfect Media Server, r/DataHoarder, the Self-Hosted podcast crowd) for exactly the mismatched-disk world we’re heading into. I want it on your radar before you need it, not after.

SnapRAID computes parity across drives of different sizes and ages, so you can add whatever disk you scrounged this month and still survive a failure or two (it scales up to six parity disks if you want the headroom). Pair it with mergerfs and those odd-sized disks present as a single pool. Honest scope: this is snapshot-style parity synced on a schedule, not real-time RAID, so it’s built for data that doesn’t change by the second; media, backups, and archives are the sweet spot. Don’t drop a busy database or a hot VM datastore on it and expect RAID behavior. For bulk storage you want to keep alive for years on hardware you already own, it’s hard to beat. It also pairs cleanly with the eBay-and-government-surplus path Eric recommended on the Vault episode, since the whole point of that path is ending up with a stack of drives nothing else can pool.

Quick Win of the Week

Find out how much life your SSDs and NVMe drives actually have left, before the market forces the decision for you. On Linux, install smartmontools and run sudo smartctl -a /dev/nvme0 (swap in your device); read Percentage Used and Data Units Written for NVMe, or the Wear_Leveling_Count and Media_Wearout_Indicator attributes on SATA SSDs. On Windows, CrystalDiskInfo shows the same health and total-bytes-written data in a couple of clicks. Five minutes per box tells you which drives have years left and which ones you should quietly start planning around while you still can.

Fun Retro SysAdmin Fact

The RAM in 1960s mainframes was literally woven by hand: tiny magnetic ferrite rings threaded onto crisscrossing wires, often assembled by women hired for the fine motor skills the work demanded, and it’s why we still say a crashed program wrote a “core dump” and why some old-timers still call memory “core.”

Until Next Week

The cheapest compute you’ll own for the foreseeable future is whatever is already humming in your rack; treat it accordingly.

Stay Frosty,

Andy

SysAdmin Weekly

• The AI doom narrative has hit saturation. Some of the doom is warranted. A lot of it is marketing.

• “AI” is doing PR work for layoffs that were really overhiring corrections. The investor signal is the point.

• Episode 046 of the podcast is out: the honest case for Claude Code in SysAdmin workflows, plus three fresh Linux LPE CVEs.

• Tool of the Week: Ollama. Run open-weight models locally, no data leaves your network.

• A full data breakdown of the doom narrative is coming on SysAdmin Weekly Podcast Episode 047 with Eric Siron in a day or two.

From the Console

The AI doom narrative has hit saturation. You feel it on LinkedIn. You see it in every other headline. Microsoft’s AI chief telling lawyers, accountants, and marketers they have eighteen months. Big tech CEOs floating that AI will replace half of white-collar work by 2030. Investors clapping. The rest of us reading the room.

Some of the doom is real. The transition is going to be painful for real people. Data center water usage today is a problem. Real jobs have been cut and real households are absorbing that. I am not minimizing any of it.

The rest of the doom is marketing dressed up as inevitability. And it is loud enough that even people who already do this work for a living are starting to second-guess themselves. That is the part that bothers me.

So I went looking at the actual data. Layoff numbers. The WEF Future of Jobs Report. Energy and water projections. What the hyperscalers are actually doing about it versus what the headlines say they are doing. There is a full breakdown coming on the podcast with co-host Eric Siron in the next day or two. Until then, the short version: the data does not say what the headlines say it says.

The rest of this issue is about the part of the doom narrative that does not get enough scrutiny in our circles: how it is being used.

And now, back to our regularly scheduled programming.

The Latest on the SysAdmin Weekly Podcast

Episode: 046 - Can Claude Code Help SysAdmins? Scripting, Log Analysis, and the CLAUDE.md Workflow

Topic: Andy makes the honest case for Claude Code in SysAdmin workflows, with the caveats and the parts that still fall short. Plus three recent Linux kernel local-privilege escalation vulnerabilities and what they mean for your patch cycle.

Reasons to tune in:

• Most AI demos are built for developers. This is a long-time SysAdmin walking through what actually helps and what does not

• The CLAUDE.md workflow is the single biggest thing most folks are missing when they try AI in their day job

• Three Linux LPE vulnerabilities (Fragnesia, DirtyFrag, CopyFail) dropped in quick succession. Local-only, but still on the patch list

If you prefer to read about my experiences with Claude Code instead of the podcast episode, I did a write-up over on my blog at andyontech.com

All Links For the Show: https://www.sysadminweekly.com or watch / listen below!

The Take

The piece of the AI layoff story that does not get enough airtime is the part where AI is the excuse and not the reason. Again, not downplaying those jobs that were legit replaced by AI, but the enterprise space seems to love making AI the scapegoat, warranted or not.

The early wave of 2023 tech layoffs had a different reason on the slide deck. IBM and a few others actually said it: we over-hired during COVID by something close to thirty percent, the demand correction hit, and we cut. That is not a comfortable thing for an executive to say in public. It also is not an investor-friendly thing to say at an earnings call. So the language quietly changed.

By 2024 and into 2025, the same kinds of layoffs were getting a different label. “Restructuring around AI.” “Reallocating capital toward AI initiatives.” “Aligning our workforce with the AI-first future.” Conveniently, that wording does two things at once: it makes layoffs feel like strategy instead of correction, and it signals to investors that the company is an AI play, which moves the stock.

We are not just speculating here. Cisco cut roughly 4,000 people on the back of strong earnings. Oracle is reportedly cutting up to 30,000 jobs (about 18% of its headcount) while standing up a massive data center buildout. Meta announced 8,000 more cuts in May while telegraphing $115B to $135B in 2026 capital expenditure, almost all of it AI infrastructure. Read those press releases against each other and the pattern is consistent: trim the headcount line, route the savings into AI capex, take the investor bump for being “AI-first.”

Again, that doesn’t mean AI is NOT displacing some jobs. It is, in specific roles, at specific companies, on a real timeline. It does mean, though, that “we cut X people because of AI” is doing a lot more PR work in 2026 than it is doing operational work. And if you are a SysAdmin, an MSP operator, or an IT manager trying to read the temperature in your own organization, that distinction matters. The internal memo and the external press release are not telling the same story for the same reason.

The piece I keep coming back to is something Eric Siron mentioned in a recent recording of the podcast….. when an executive who has never been an accountant tells accountants they have eighteen months, ask that executive what THIER numbers actually need to look like by then. Eighteen months is often less about when the technology will be ready and more about when the latest AI bet has to start showing returns.

We will get into the data side, the WEF Future of Jobs numbers, the historical tech wave precedents, the water and power question, in detail when Episode 047 drops with Eric Siron. For now: if your CIO walks in citing the eighteen-month deadline, ask them to source it. Real conversations get easier from there.

Community Signal

High-signal community work worth your attention this week.

Brandon Lee (VMware vExpert) - “I Built a Local AI Coding Agent Home Lab Setup With OpenCode and Ollama” - The practical antidote to the doom narrative… practical use of AI systems! In this post Brandon Lee walks through standing up OpenCode against a local Ollama instance, with the model running on his own hardware and no data leaving the lab. The part worth the read is the honest scoping: where the local setup actually competes with the cloud offerings, where it does not yet, and what hardware you need to make it usable. Practitioner work, no vendor influence.

Tool of the Week

Ollama - Run open-weight LLMs locally. Single binary, simple model pull syntax, OpenAI-compatible API surface for any tooling that already speaks that protocol.

The pitch in 2026: if you are tired of either feeding sensitive data to a hosted model or being told you are not allowed to use AI at all, Ollama is the third option. Pull a model (Llama, Qwen, Mistral, DeepSeek, Codestral, others), point your client at the local endpoint, and run inference on your own hardware. Pairs cleanly with OpenCode, Continue, and any IDE plugin or CLI tool that supports a custom OpenAI-compatible base URL.

To be straight: the local model quality at consumer hardware tiers is not the frontier-model quality you get from Anthropic or OpenAI. For privacy-sensitive workflows, internal-only use cases, log triage, and scripting against data you cannot ship outside the network, the gap closes considerably. For greenfield engineering work, you will still reach for the hosted option more often than not based on my experience. Either way, nice to have the options.

Quick Win of the Week

Pick one “AI is replacing X” headline you have seen this month. Trace it back to the primary source. Is it a CEO quote, a press release, a paper, or a survey? Whose survey? How were the numbers gathered? You will be surprised how often the headline number does not survive the trip back to where it came from. Sourcing your own claims is one of the most useful skills a SysAdmin can keep sharp in 2026.

Fun Retro SysAdmin Fact

The first “AI is coming for your job” panic in IT was not in 2023. It was in the late 1980s, during the expert systems boom, when symbolic-reasoning platforms running on Lisp machines were supposed to replace whole categories of decision-making work. By the early 1990s the entire commercial expert systems market had collapsed into what historians of computing now call the second AI Winter. The Lisp machines went to museums, the panic dissipated, and the next wave of IT growth (the consumer internet) created more jobs than the AI Winter ever displaced.

From My Other Corner of the Internet

AndyOnTech

Can Claude Code Help SysAdmins? Real Use Cases for IT Pros - The companion write-up to Episode 046. Covers the CLAUDE.md workflow, scope-based settings, and where to stay skeptical with sensitive data and production-adjacent prompts. If the episode landed, the post is the version you can hand to a teammate who prefers reading to listening.

Thing I’m Re-Thinking

The assumption: When a major tech CEO makes a specific timeline claim about AI displacement, I should weigh it as informed analysis from someone close to the technology.

Why it no longer holds up: After a few iterations, it REALLY seems like a lot of those claims are not predictions, they are deadlines. Eighteen months is a number that lines up with when AI infrastructure spend has to start producing returns. The person saying it has a financial reason for that number, and that reason is not “this is the date the technology will be ready.”

What I am testing instead: Treating those statements as investor communications, not as forecasts. If the same claim comes from someone who has actually done the job being replaced, who is working at the keyboard with the tools, and who has no AI-adjacent stock price to defend, that is the version I weigh.

Until Next Week

If the doom is loud enough that you are starting to repeat it without checking, that is the moment to go check.

Stay Frosty,

Andy

SysAdmin Weekly

• Was at a conference recently. Half the room was on their phones. That’s not a people problem.

• The culture of IT quietly starves curiosity. And we as SysAdmins let it happen.

• I stood up Forgejo in my lab out of curiosity. It ended up replacing every GitHub component of my workflow. Zero regrets.

• Lab work isn’t a hobby. It’s how you stay dangerous.

• This week: do something that doesn’t have a ticket number.

From the Console

I attended a recent tech event in Texas which turned out to be a good event, had decent content, and a strong sense of community. At one point, I’m sitting there looking around the room….observing people as I’m want to do at times to gauge interest in a particular topic. I noticed lots of phones out, laptops open, heads down and my initial reaction was frustration, because I’ve been the guy up on stage talking, many times. Then I caught myself.

These aren’t checked-out people. These are DEPLETED people. The queue never stops. The alerts never stop. And somewhere in all that noise the thing that made most of us good at this job (the curiosity), the “I wonder what happens if I press this button” instinct, just quietly gets starved out.

That’s the thing worth talking about this week.

Folks that are regulars of the SysAdmin Weekly Podcast have likely heard me talk about the importance of curiosity for SysAdmins. I’ve been in the industry long enough now to know that SysAdmins used to have more time to be curious. We used to have more time for training. In fact, we’ve done our jobs so well over the last decade or two that tech and “digital transformation” have become core to many businesses. So much so, that the many businesses simply cease to function when tech systems are offline or impacted in some way.

Pair that with the fact that many organizations still view IT as a cost center. I still run into accounting heads that see IT as a black hole for money, and I still see leadership teams not giving IT teams the resources or the time needed to get the job done while leaving room for self-improvement or self-care. SysAdmins are overworked, and understaffed / underfunded, and it shows.

That all said, I would urge all of you to maintain that curiosity. Tinker in the lab. Check out that new community tool. Spin up that new game server. Most of us got into this industry because we have a deep love for the technology. Sometimes experimenting with new tech for no reason other than curiosity is all that’s needed to keep the bad parts of the job from overwhelming you and in the process it helps you keep your skills current and growing.

And now, back to our regularly scheduled programming.

The latest on the SysAdmin Weekly Podcast

Episode: 045 - Why Is It Always DNS? How DNS Actually Works, Breaks, and Gets Exploited Topic: Andy and Eric pull apart the full DNS resolution chain, Active Directory DNS integration, DNS security threats, and two war stories that will make you go check your fourth domain controller.

Why this one matters: - Most SysAdmins work with DNS every day without actually understanding how it works. This is the episode that fixes that - If you’ve ever said “have you tried flushing the DNS cache?” and quietly hoped it worked, this one is for you - DNS is also how attackers move data and talk to compromised machines. Understanding the protocol means understanding the threat

Episode Links

All Links:

https://www.sysadminweekly.com

YouTube:

Spotify:

The Take

I stood up Forgejo in my homelab recently. Self-hosted git forge, two-class repo model, nginx reverse proxy, internal HTTPS with mkcert. Did I plan for this to become my full git workflow? No. It started as curiosity. It ended up replacing every GitHub component of my day-to-day. GitHub no longer has a seat at the table, unless a repo has to be public and in that case I use repo mirroring so I still only have to interact with Forgejo at the end of the day.

On top of those benefits the curiosity that drove me to test out Forgejo to begin with enabled me to learn things I wouldn’t have learned any other way.

That’s the whole point. The lab isn’t always about the output. It’s about keeping the instinct alive. The instinct to dig in, break something on purpose, and figure out why it works. That muscle atrophies when you stop using it… and the environment most of us work in is perfectly designed to let it atrophy. Tickets, queues, SLAs, “what did you close today.”

It’s become a sad reality that the people who think lab work is a waste of time are usually measuring the wrong thing. They want deliverables. The lab doesn’t produce deliverables. It produces practitioners who don’t panic when something breaks outside the documentation.

This isn’t a generational thing either. I’ve seen 20-year veterans lose the curiosity instinct just as fast as someone two years in. It’s a culture problem. The fix is the same regardless of time in the chair: build something this week that nobody asked you to build. I’ll be money that two things will happen:

1. You’ll learn something new

2. You’ll enjoy the time spent doing so

So that said, what personal tech project are you interested in that you’ve been putting off? Maybe today is the day to start?

Community Signal

High-signal community work worth your attention.

mrlokans.work - “State of Homelab 2026” - A personal, opinionated stack writeup from someone who documents their gaps honestly: no backups, single NVMe, running costs under $10/month. The part worth reading is the SOPS with age encryption section as an upgrade over Ansible Vault for version-controlled secret management. Good real-talk energy, no vendor influence, highly recommend!

Brandon Lee - “How I Built a Self-Healing Home Lab That Fixes Itself” - Brandon Lee documents building an n8n automation pipeline that monitors CI/CD via Discord, uses an AI agent to identify failures, and automatically re-triggers failed pipelines without human intervention. Proxmox, Prometheus, Grafana, Docker restart policies. The interesting angle here isn’t the specific stack; it’s the idea that the lab itself can model the same automation discipline you’d want in production. Also a good read for this week!

Tool of the Week

Beszel - Lightweight infrastructure monitoring with a hub-and-agent model that actually fits the way homelabs and small shops are built.

Lightweight single binary, hub-and-agent model. Monitors CPU, memory, disk, network, and Docker/Podman container stats. The web dashboard is clean, alerting is configured with sliders and toggles instead of a rule language, and it doesn’t require standing up a full Prometheus stack to get meaningful visibility.

To be straight with you: this is a homelab and small fleet tool. No SNMP, no custom metrics, no HA, SQLite-backed. It is not a Zabbix or Prometheus replacement for enterprise environments with SLA requirements. For everything else, it’s the current community default for a reason.

Quick Win of the Week

Pick one service you interact with every day and actually read how it works this week. Not a tutorial. Not a YouTube walkthrough. The documentation or the RFC. Twenty-five minutes. The goal isn’t to become an expert; it’s to make contact with the actual thing instead of just operating the surface of it.

Fun Retro SysAdmin Fact

Before DNS existed, every host on the ARPANET was tracked in a single HOSTS.TXT file maintained by one person at the Stanford Research Institute and distributed via FTP. By 1983, the growth of the network had made manual updates impossible, which is what prompted Paul Mockapetris to design DNS. The specs he wrote in RFC 1034 and RFC 1035 are still the foundation of every DNS lookup happening right now.

Worth Your Time

Forgejo - User Documentation - If The Take in this week’s edition landed and you want to actually spin up your own git forge, the docs are legitimately good. The user docs index includes a getting started section.

Uptime Kuma - If you spin up Beszel, pair it with this. Beszel watches internal resource metrics; Uptime Kuma watches external service availability (HTTP, TCP, DNS, SSL). They cover different failure modes and the community runs them together for that reason. Worth mentioning that it also has a large number of GitHub stars….. meaning this one has been around long enough to trust.

Thing I’m Re-Thinking

The assumption: Staying current means following the news cycle.

Why it no longer holds up: Reading about technology and understanding technology are not the same thing. You can follow every CVE, every product announcement, every LinkedIn hot take… and still be completely lost the first time you have to operate something unfamiliar under pressure.

What I’m testing instead: One new thing in the lab per month. Doesn’t have to be big. Doesn’t have to be relevant to the day job. Just has to be something I haven’t run before.

Until Next Week

The queue will be there Monday. Your curiosity might not be if you keep telling it to wait.

Stay Frosty,

Andy

SysAdmin Weekly

· Big Tech is consolidating compute, cloud, and client hardware at an unprecedented pace and open source is increasingly the only viable counterweight.

· The Steam Deck is the clearest mainstream proof that open source can compete on quality, flexibility, and user freedom, even in a consumer market dominated by locked ecosystems.

· Open source isn’t just idealism. It’s a practical, career-relevant strategy for SysAdmins managing costs, avoiding lock-in, and maintaining operational control.

· This week’s Community Signal highlights open source tools worth building into your stack right now.

· Tool of the Week: A self-hosted alternative that punches well above its weight class.

From the Console

The Locked Room Problem

I want you to think back to a time… maybe recent, maybe not, where a vendor changed something fundamental about a tool you depended on. Maybe it was a licensing model. Maybe it was a feature that got paywalled. Maybe it was an acquisition that turned your go-to platform into a product roadmap you no longer recognized.

The most recent example for me, was Firefox. And if you’ve been listening to the podcast or have read my lengthy blog post on the subject, you already know how I feel about that particular saga.

That thing I keep coming back to: that feeling of helplessness….of being locked in a room where someone else controls the door, isn’t unique to Mozilla. It’s, sadly becoming the default state of modern enterprise IT, for a number of reasons.

In issue #21, we talked about the Great Compute Divide. Big Tech is hoarding GPUs, cloud vendors are pitching you on renting your own PC BACK from them, and RAM prices have quadrupled because the market for on-prem compute is being systematically starved (whether that’s the intention or not remains to be seen). That said, the consolidation is real, it’s accelerating, and it has consequences.

So what’s the counterweight? Despite my example above being Mozilla / Firefox (which has started catering to market demands) I’d argue that the long-term counterweight is in fact open source. It’s also worth noting that I don’t mean that in the idealistic, “free software should be free” sense (though that argument has merit). I mean open source as a practical, professional survival strategy for SysAdmins.

And if you want the clearest mainstream proof that this strategy works, look no further than Valve’s Steam Deck.

Here’s a consumer device running a custom Arch Linux distribution called SteamOS that ships with full terminal access, supports arbitrary software installation, and lets you configure, break, and rebuild it however you see fit. Valve didn’t lock it down. They handed you the keys. And the result? A device with one of the most enthusiastic and capable user communities in consumer tech, running games and workloads that “shouldn’t” work because the platform respects user freedom.

That’s the open source promise made tangible…. And it’s exactly the philosophy SysAdmins need to be applying to their own stacks right now.

This is the part where you all come at me with torches and pitchforks… “But, Andy! What about support”? I would make the argument that the support question can be worked around while vendor lock-in, hardware scarcity, planned obsolesence, and adversarial behaviour towards customers are much more difficult to contend with. At least with the support question you can:

· Train up internally

· Partner with a third party

· Some Linux distros have their own support mechanisms

The other issues I laid out above have more difficult solutions for the most part.

Whatever your stance on this issue is, I would urge you, at the very least, to conduct an audit of all functions within your environment where you depend solely on a single vendor. Which of those vendors have you locked in, in some way? Those are the areas you need to at least have a fallback plan in mind in the event that things go south.

A SysAdmin Clinging to an Open Source life preserver

And now, back to our regularly scheduled programming.

🎙 Recently on the SysAdmin Weekly Podcast

SysAdmin Weekly - 039 - BitLocker, Key Escrow, and the Microsoft Trust Question

All the relevant links for this episode can be found below:

· SysAdmin Weekly Website

· SysAdmin Weekly Github Discussions Board

The Case for Open Source in 2026 - Beyond the Ideology

I’ve been on a practicality kick lately with my content. So, let’s get practical for a second. Building on the “From the Console” section in this week’s issue of the newsletter, here’s why open source deserves a strategic seat at the table for every SysAdmin today:

1. You Own the Exit

Proprietary software comes with a quiet hostage clause, which is sometimes spelled out (and sometimes not). When the vendor raises prices, discontinues a feature, gets acquired, or simply decides your use case no longer aligns with their roadmap, you are stuck….full stop. Open source breaks that clause. You can fork it, self-host it, freeze it at a version that works, or migrate on your terms.

This isn’t hypothetical. Looking at a few examples:

· BookStack - Open Source Wiki / Documentation App

· Gitea - Self Hosted Open Source Code Repo Software (Note: owned by a for-profit org)

· Forgejo - Self Hosted Open Source Code Repo Software

· Nextcloud - Self hosted cloud filestorage

· Restic - Awesome CLI-based backup application

These are all tools that we’ve mentioned in this newsletter before and they exist because the proprietary alternatives became untenable for a meaningful number of organizations. The community voted with their forks.

2. Auditability Is a Security Feature

Putting my security brain in place for a second, I would also make the claim that, in general, you cannot fully trust what you cannot inspect. There are some exceptions to that rule for certain software suites, sure, but I feel it’s true when speaking generalities. The fact is, that with proprietary tooling, you’re accepting a black box at the foundation of your infrastructure. With open source, the code is accessible and viewable. Security researchers can (and do) find and fix vulnerabilities more quickly because the surface is visible. Now, it’s worth noting that this isn’t always comfortable. Visibility cuts both ways, but at least it’s honest, open, and everyone can see things for what they are, warts and all.

3. Cost Control in a World of Subscription Creep

SaaS subscriptions compound quietly. Open source tools, even with self-hosting overhead, frequently deliver better TCO at scale, especially for non-glamorous infrastructure services: documentation platforms, backup tools, monitoring stacks, identity providers.

4. The Talent Pipeline Recognizes It

The next wave of SysAdmins grew up on GitHub, Linux, and Docker. Open source fluency is increasingly a hiring signal, and building an open-source-forward environment makes your org more attractive to the people you want to retain.

Core Fundamentals: Reading an Open Source Project Before You Commit

Not all open source is created equal. Before you adopt a tool and build workflow around it, here’s a quick due diligence checklist:

· Commit frequency: Is the project actively maintained? When was the last commit?

· Issue response time: Are maintainers engaged with bug reports? Or are issues piling up unanswered?

· License: MIT, Apache 2.0, and GPL all have different implications for commercial use and modification. Know which one you’re working with.

· Community size: A project with thousands of GitHub stars and an active Discord/forum is less likely to be abandoned than a solo maintainer’s side project.

· Commercial backing: Projects with a company behind them (HashiCorp, Grafana Labs, Canonical, Gitea) have longevity. That said, watch for the “open core” model that some of them offer. Open core is where the good stuff is still generally paywalled.

· Documentation quality: Poor docs are a maintenance burden. Great docs are a force multiplier.

Community Signal

High-signal community content worth your attention this week > open source edition.

awesome-selfhosted – The Community’s Master List If you only bookmark one link from this edition, make it this one! Awesome-selfhosted is a community-maintained GitHub repository cataloging hundreds of free, open source network services and web applications you can host on your own infrastructure. Organized by category, it includes stuff like: analytics and backup tools to document management, monitoring, and identity providers…etc..etc. It’s the definitive reference for SysAdmins evaluating self-hosted alternatives to SaaS platforms. Think of it as the antidote to vendor lock-in, one category at a time.

Gitea Official Migration Documentation Full disclosure: There have been some community concerns around Gitea now existing under a for-profit company (See further down re: Forgejo). That said, Gitea remains a solid git repo hosting option that still retains the open source label. This documentation is a great resource for getting started if you’ve never used Gitea before. The official documentation covers installation, systemd service configuration, access, etc. That migration angle is the key differentiator here. It’s not just “here’s how to install a thing.” It’s “here’s how to actually move the thing.” If you’ve been meaning to get your repos off a platform you don’t control, this is your starting point.

TechReviewer – Linux Hits 90% Windows Game Compatibility Milestone The Steam Deck and Proton keep coming up in this edition for good reason. They’re the clearest mainstream proof that open source can compete at scale without compromising on user experience. This piece puts hard numbers to that argument: over 5.6 million Steam Decks sold by mid-2025, 21,694 Deck Verified games, and nearly 90% of Windows titles now running on Linux via Proton. The story here really shows what happens when a platform holder commits to openness and invests in the community infrastructure to back it up. Sound familiar?

Tool of the Week

Forgejo - A community-driven fork of Gitea, born after concerns about Gitea’s governance direction under a new for-profit organization. Forgejo is a fully self-hosted Git service. Like Gitea, you can think of it like GitHub, but it’s yours and it lives in your environment. Lightweight, fast, and increasingly the preferred choice for SysAdmins who want source control without handing data to Microsoft.

Quick Win of the Week

Audit one proprietary tool in your current stack this week. Ask these three questions:

1. What does it cost annually (licensing + admin overhead)?

2. Is there a credible open source alternative?

3. What would migration realistically take?

You don’t have to move anything. Just know your options. Because the time to evaluate alternatives is before your vendor sends you a renewal notice with a 40% price increase.

Fun Retro SysAdmin Fact

The GNU Project, which is the foundation of modern open source as we know it, was announced by Richard Stallman in September 1983. That means the open source movement is older than most of the Windows Server versions you’ve probably had to decommission. And unlike those servers, it’s not going anywhere.

Until Next Week

The tools to stay independent already exist. They’re battle-tested, community-supported, and increasingly production-ready. The question isn’t whether open source is viable, the Steam Deck is one project proved that argument is over, in my opinion. The question is whether you’re building your stack in a way that keeps the exit door unlocked.

Thanks for reading and stay frosty out there!

Andy @

• Big Tech is consolidating massive amounts of compute while AI workloads continue driving unprecedented demand for GPUs and memory.

• Major cloud vendors are openly advocating for a future where you rent your PC and infrastructure instead of owning it.

• RAM prices have exploded, with some configurations quadrupling in cost, effectively pricing out homelabs and even certain enterprise deployments.

• The more workloads move to centralized cloud platforms, the less independent control organizations retain over their own compute.

• This week’s From the Console asks a hard question: does compute consolidation ultimately reduce flexibility, choice, and long-term autonomy?

• Community signal worth your time: a practical guide to estimating and managing Azure costs before they spiral out of control.

From the Console

I think it finally happened….. I’ve finally circled back to the portion of the industry that prefers to run (most) workloads locally as opposed to “cloud-first”.

Trust me… this comes as much as a surprise to myself as it does you (or maybe not?). It was a very specific line of thinking and industry events that led me back down this path… events which we plan to cover in a future podcast episode. That said, I thought articulating them here would make for an interesting “From the Console” opener for this week’s Newsletter.

So what led me down this path? A number of different news stories over the past couple of months eventually led to a thought experiment… which, in turn, led to my new stance, I suppose.

So, what were those news stories, you ask?

Big Tech Helping Trump & GOP in the 2026 Midterm Elections

https://www.washingtonpost.com/technology/2025/12/19/tech-super-pacs-midterms-ai/

Now, first off, this newsletter is NOT a political newsletter. That said, we have mentioned politics on the podcast from time to time as it relates to the technical topics of the day. Regardless of your stance on the current state of US politics, the important and relevant bit for this discussion, is this personal take I put together re: the article:

There are a group of individuals with direct or indirect ties to “Big Tech” that are using their resources to influence a specific result(s) in the US 2026 Midterm election. When the same entities that control massive compute resources also shape political outcomes, it raises questions about long-term digital independence.

Political activism by big tech isn’t a new thing. It’s been going on for years now, but it’s an important item to point out as currently happening in our industry and it’s important to take into account for this editions “From the Console” discussion.

Quote from Jeff Bezos on Local Compute

https://www.windowscentral.com/artificial-intelligence/jeff-bezos-says-the-quiet-part-out-loud-bezos-envisions-that-youll-give-up-your-pc-for-an-ai-cloud-version

I came across this article from Windows Central in my usual RSS meanderings and Bezos’s stance on local compute is what caught my eye. This section of the article specifically.

So, what prediction did Bezos make back then, that seems particularly poignant right now? Bezos thinks that local PC hardware is antiquated, and that the future will revolve around cloud computing scenarios, where you rent your compute from companies like Amazon Web Services or Microsoft Azure.

There has been talk for some time now from the major cloud vendors talking about the “benefits” to “renting” a PC in the cloud. In fact Microsoft has even produced a line of endpoints (which are basically thin-clients) called Windows 365 Link who’s sole purpose is to connect to a cloud-hosted PC in Azure.

We’ve seen the massive consolidation of server-related workloads being pushed into large cloud environments for the last 10 to 15 years. Now, it seems, there is a renewed push to get a percentage of client market share hosted in large cloud providers as well.

Well Known and Ongoing Industry Memory Scarcity

https://tech.yahoo.com/computing/articles/ram-prices-exploding-heres-why-111115489.html

It’s become well known throughout our industry (and amongst the gaming community as well) that if you need to buy memory, do it NOW. Prices have been rising for some time and I can speak from personal experience on this one. About a year ago, I had built a decent box for some homelab workloads that has 96GB of memory in it. Out of curiosity, I checked what that same 96GB of memory would cost me today and I discovered that the cost for that given configuration has QUADRUPLED in price. I’ve even heard stories from fellow enterprise admins that their organizations have been effectively priced out of certain configurations due to the massive cost increase of memory.

We’ve seen the industry’s push to leverage cloud compute more and more. That said, there has always been a cohort of SysAdmins that have either been unable, or simply do NOT want the loss of control that using someone else’s hardware entails.

What Concerns Does This Raise?

Having consumed all three of these news stories in close succession, my brain started something of a thought experiment with some key points:

· The current industry (and arguably geopolitical) line of thought is: AI = Inferencing = Power

· We know that AI workloads require a gargantuan amount of compute resources

· There has been increasing market inertia to move workloads to the cloud

· Increased cloud dependence REMOVES the need for on-prem / owned compute hardware

This pattern ultimately takes us to one place, and that is a world where the hardware needed for massive compute workloads (like large-scale AI) is largely in the hands of a few major players.

The question I leave you with today is this:

Does the consolidation of compute, and by extension, the ability to run compute-heavy workloads (like AI), eventually take away choice, flexibility, and independent operation absent influence from big-tech & politics? And, does this sequence of events take away power from individuals by extension?

Also worth asking:

What happens when access to high-performance compute becomes a subscription privilege instead of an owned capability?

We’ll be sure to cover this in a future episode of the podcast with more discussion. However, for now, I’d LOVE to hear what your thoughts are on this! Feel free to comment below!

Big Tech Hording All the Compute

And now, back to our regularly scheduled programming

🎙 Last Time on the SysAdmin Weekly Podcast

Episode: SysAdmin Weekly - 038 - Making Security Decisions Based on Data, Not Fear

SysAdmin Weekly - 038 - Thumbnail Image

Topic: Given the state of tech marketing and the Cybersecurity news-cycle currently in the industry, this topic is increasingly important for today’s SysAdmins. It’s easy to fall into the trap of FUD (Fear, Uncertainty, Doubt) when it comes to Cybersecurity. The reasoning is rather simple. While we SysAdmins know that security is a necessary component of any modern network, the psychology behind the NEED for security comes from a place of danger and uncertainty. We must protect our networks against threats both real and suspect, so when a vendor comes knocking, saying “What if?”, it’s easy to to fall down that rabbit-hole.

In this episode Paul and Andy discuss steps you can take to make sure that you’re making these choices from a place of data and setting the fear and emotion to the side. Links to the applicable platforms can be found below:

· SysAdmin Weekly Website

· SysAdmin Weekly - 038 - on YouTube

· SysAdmin Weekly - 038 - on Spotify

· SysAdmin Weekly - 038 - on Apple Podcasts

Also, if you’ve got thoughts to share on this episode be sure to join the SysAdmin Weekly GitHub Discussion Boards!

Community Signal

I’ve got one Community Signal item for readers this week, and it’s high quality.

Thomas Maurer - Azure Cost Estimation

Beings compute consolidation and big public cloud platforms were a big discussion earlier in today’s edition, this resources felt applicable. I know MANY SysAdmins that are dealing with cost management on public cloud platforms and it can be really difficult to get an accurate estimation of your cloud-spend, especially when rolling out new services.

Thomas Maurer always provides fantastic resources and this post is no exception. I highly suggest you check it out!

A Backup Tool Worth Your Time

Restic is an open-source, cross-platform backup tool designed for simplicity, security, and efficiency. It performs fast, incremental backups using content-defined chunking and deduplication, meaning only new or changed data is stored after the first run. All backups are encrypted by default using strong cryptography before leaving your system, so even the storage provider cannot read your data.

Restic supports a wide range of backends including local disks, SFTP, SMB, and cloud object storage such as S3-compatible services, Azure Blob, and Backblaze B2. It is lightweight, script-friendly, and reliable, meaning it’s AWESOME for SysAdmins and homelab enthusiasts.

Official documentation is available HERE

From AndyOnTech

Why You Should Consider Debian Linux in 2025

I’ve spoke about this particular post in the newsletter in the past, but I’ve seen a number of “Why Debian” related posts on Reddit as of late, and even ran into the question on social media in the last week or so as well.

I wrote this article last year, so it’s a bit dated, but all the reasons why you might want to consider Debian Linux all stand in 2026. If you want the gritty details (and a YouTube video!), you can check out the link. Otherwise the short list is here:

· Stability

· Customizability

· Free and Open Source

· Highly Secure

· Fantastic Community

· Excellent Server Performance

· Wide Hardware compatibility

· Long-Term Servicing Options

If you find yourself currently hopping around and testing distros, I highly suggest you consider Debian. New features to do lag from time to time, but Debian can’t be beat in terms of rock-solid stability!

Until Next Week

I want to make sure I say it again. We here at SysAdmin Weekly REALLY appreciate all of you that partake in our content and community. As this weekend brings us the Superbowl in the US, I wish you a quiet weekend and may all your systems stay running, at least until the big game is over.

Until next week, stay frosty!

— Andy @

• Incident response plans and backup / DR strategies are boring, neglected, and absolutely career-saving when things go sideways. Review them before you need them.

• New podcast episode out: When Incident Response Plans Meet Reality, covering what actually belongs in an IR plan, who should be involved, and how tabletop exercises should work in practice.

• Curiosity isn’t optional for SysAdmins. Using short, intentional learning sessions can help you keep skills sharp even when motivation is low.

• Community signal worth your time: strong arguments for using Markdown and the CLI to improve focus, consistency, and long-term maintainability of technical documentation.

From the Console

With it still being somewhat near the start of the year, we’ve been reminding our listeners that it’s a good time to take a look at those…. foundational SysAdmin things that we tend to neglect. This would be things like Backup & DR planning, incident response plans…. recovery testing…etc.

Now to preface this opening section of this week’s newsletter I’m not judging anyone here. I get it. End-users are pounding on your door, mail services went offline AGAIN, and you’ve got a project list as long as the help output for ffmpeg. The last thing you want to think about are incident response plans that should still be good.

Also relevant along these lines and interesting, is our watch / listen metrics for last week’s episode aligns with this mindset as well. In our episode last week Eric and I discussed Incident Response Plans in the Real World and the consumption of that episode was noticeably less than normal. Again, I’m not knocking anyone, it’s just an interesting data point that supports my theory that the average SysAdmin either doesn’t have time for incident response / backup & DR planning or its boring and / or overdone in terms of content, or everyone assumes existing plans are…. fine.

I have to say, that in terms of interest and “general IT priorities”, incident response planning isn’t the sexiest topic, but it is a critical and foundational one. As SysAdmins, it’s our job to insure systems uptime and data safety whether that (thankless) work is visible to the organization or not. An (at minimum) annual review of your incident response and backup & DR plans does a couple things:

· Highlights technical gaps or limitations within critical business infrastructure

· Informs key stakeholders within the business that may have moved / changed / forgot about these initiatives

· Helps prevent overly long outages

· Safeguards company data

· Makes you look like the superhero and INSTANTLY validates your salary the second something happens and you’ve got the magic plan that saves everything

What’s the saying?

“An ounce of prevention is worth a pound of cure.”

Block some time off in the calendar, and soon. Again, I know these items aren’t interesting. They aren’t some shiny new tech that needs to be installed. They aren’t business transforming things, but you’ll be much better served to have them ready (and updated) and not need them, than the other way around. Chaos without a plan is just chaos. Chaos with a plan is still chaos, but at least it’s managed chaos where you come out the hero, and your organization stays out of news.

If you’d like some real world guidance on the review of your incident response plan and backup / DR plan, be sure to check out last week’s episode of the SysAdmin Weekly Podcast listed below!

Also, if you’d like to chime in with your incident response / backup & DR success (or horror) stories, check out this thread on the SysAdmin Weekly GitHub Discussions Board!

A SysAdmin ignoring the incident response plan in favor of cool infrastructure projects

And now…. back to our regularly scheduled programming.

🎙 The latest on the SysAdmin Weekly Podcast

SysAdmin Weekly - 037 - When Incident Response Plans Meet Reality

SysAdmin Weekly - 037 - When Incident Response Plans Meet Reality

Topic: In this week’s episode Andy and Eric discuss one of the most groan-inducing and seemingly boring SysAdmin Topics on the planet: Incident response plans. While many SysAdmins dislike the documentational nature of building (and maintaining) incident response plans, they are one of the most critical components to ongoing operations. To put it shortly, when you need it, you NEED it.

This episode gives you the details on:

· What to include in your incident response plan

· What stakeholders within your organization to involve

· Frequency and details covered during table-top exercises

· Lots more!

The episode also includes the usual nerd-hour talk, VMware / Broadcom gripes…etc.

Episode Links Here!

· All platform links

· SysAdmin Weekly - 037 - On YouTube

· SysAdmin Weekly - 037 - On Spotify

· SysAdmin Weekly - 037 - On Apple Podcasts

The Take - Curiosity Should be a Weekly Mandate for SysAdmins

I wanted to make a short take this week to continue some commentary on the importance of curiosity as a core trait(s) needed by SysAdmins.

It’s easy in this industry to rest on your existing knowledge, but you’ll find that you fall behind pretty quickly. It can be difficult to focus on learning new skills on the regular, especially when it’s (maybe) a technical skill that you’re really not interested in, but is required due to things you have to support during your day job. I find that curiosity helps with the resistance in this situation.

There are certain technical skills that I’ve committed to training myself on over the course of 2026, but they aren’t always the most interesting (I’ll plan a future blog post on this). What I find works well for me is to spend a little time digging into something that I AM highly interested in prior to tackling the “un-interesting” skill. For example, I’ve long been curious about Unity. While I’m not officially a programmer, I dabble. On top of that, I’ve been playing video games now for nearly 40 years, which is basically how long I’ve been able to properly use a controller. As such, I’ve always had an interest in tinkering with game engines and maybe someday building my own game.

With that said, the sequence I’ve found that works well for learning those “un-interesting” skills is the following:

· Define a training time period

· Start with a short session tinkering / training on something that actively interests you. (Unity in my case)

· Actively work on the training / tech that you’re obligated to work on

· Repeat sequence as needed

Is this context switching? Yes… to a degree, and yes, I do hate context switching. But, session lengths should be long enough to where it doesn’t really have an impact. For example, I follow the pomodoro method for just about everything. In this context, I’ll tinker with unity for 25 minutes (one pomodoro) and then commit 2 pomodoros to the other training need.

This method helps make sure that I learn the skills that I need to learn for external reasons, while also keeping my brain engaged and in “learning mode”.

Hopefully this will be as helpful to you as it is to me!

Community Signal

High-signal community work worth your attention.

Ryan Elston – Why I Use Markdown and Why You Should Too

As I’ll shortly be singing the praises of Markdown later in this newsletter, this article by Ryan Elston seemed to be highly relevant. In this post Ryan covers why Markdown is a highly valuable documentational language and even provides a great argument for why you SHOULD be using markdown along with some markdown authoring tools to get started!

Mike Robbins - Write clear step-by-step Instructions Using Markdown

Mike Robbins is also a fantastic source of information about markdown as well as how to properly use it in a how-to format. It’s worth noting that Mike is also a great source of PowerShell scripting knowledge as well! His blog is highly recommended!

Question I Got Asked (and the Real Answer): “Why are you obsessed with the CLI and markdown?”

Boy, it’s been a very commentary heavy week for me throughout this edition, but I feel like this will all provide some value (at least I hope it does).

Those that watch / listen to The SysAdmin Weekly Podcast know that I am an absolute CLI and Markdown nerd. My co-hosts have even made fun of me during certain episodes, which is warranted if I’m honest about it.

It so happens that I have had this question levied at me several times over the past couple of weeks:

*“Why are you obsessed with the CLI and markdown?”

First, valid question. Second, I’ll plan on a long form blog post or YouTube video (maybe both?) that explains my method and the “Why?” behind it.

In short I author ALL of my content using the following tools:

· Kitty (A terminal emulator)

· vim (A CLI-based text editor)

· Git, GitHub, & Gitea (to act as change tracking methods & content repositories)

· Markdown (my documentational language of choice)

· Pandoc (for converting markdown content to finalized formats)

To keep it short and straight to the point here, the core reasons “Why?” are as follows:

· I’m an easily distracted person, and CLI UIs are clean, focused, and purpose-built.

· Markdown is a defined and rule-driven documentational language that should output in concise formatting regardless of the author, assuming markdown principals are followed.

· As part of my job and my tech community efforts I spend A LOT of time authoring content of various types.

· A CLI environment paired with markdown prevents context switching, enables focus, and insures unified and consistent output.

· By using Git and Git-compatible repos I get the full benefit of code-review, pull requests, and commit history through the lifespan of the content

I’ll plan on covering this in a future blog post in more depth or via a podcast episode, but if you have questions you can reach out to me on LinkedIn or feel free to start a thread on the SysAdmin Weekly GitHub Discussion Board!

Until Next Week

Thanks for hanging out with us for another week! May you remain curious in your SysAdmin journey, and may your packets continue to flow as planned throughout the weekend!

Stay frosty!

Andy @

⏩ TL;DR – This Week in SysAdmin Land

• Abstraction is great… until the dashboard lies to you. A real-world Kubernetes + Longhorn storage lesson on why “schedulable” doesn’t always mean usable.

• Running stateful workloads on Kubernetes still requires understanding the underlying storage mechanics, replicas, disk sizing, and node-level constraints matter.

• SysAdmin Weekly now has an official GitHub Discussions home for announcements, technical discussions, polls, and general sysadmin banter.

• New podcast episode out: The Hidden Dangers of Abstraction in Modern IT that covers why deep technical understanding still matters in 2026.

• Community signal worth your time: a grounded take on why you probably don’t need 10 Gbps home fiber, and a smart rethink of how GitHub Copilot should actually be used.

From the Console

I was reminded to embrace curiosity in IT again this past weekend

In perfect timing with our episode regarding abstraction in modern IT (more on that below), I was reminded to embrace curiosity and not blindly trust / follow abstraction layers in modern software.

As many who watch the podcast know, I’ve been slowly converting many of the workloads in my homelab into containers running on a (now) 4-node K3S (Kubernetes Lite) cluster. This project was brought on by a number of things:

· I’ve always leveraged platforms like Azure Kubernetes Service and wanted to get up to speed on running local Kubernetes

· I’ve got aging hardware in my homelab and getting everything to follow Hyper-V best practices is…. difficult

· Distributed Architecture has always been highly interesting to me

· Many of the workloads I run benefit from being stateless

· I’ve rediscovered my love of all things open-source recently, and have taken steps to start using more stuff from the FOSS ecosystem

While, yes, many of the workloads I run can benefit from being stateless, I have several workloads that MUST be persistent, and as such, require persistent storage for data, state, etc…etc. To facilitate this in a clustered and highly-available fashion, I’ve deployed Longhorn to the cluster to provide distributed block storage across the cluster.

It’s an amazing system really. Unlike hypervisor clusters that require high degrees of compatibility checks, fabric deployment & management, etc, Longhorn makes storage configuration STUPID easy. Once setup and configured on the cluster, anytime a new node is added to the existing cluster, Longhorn will automagically ingest the available storage. Storage is spread in a redundant fashion across all the available nodes and fault domains are automatically defined and configured. Abstraction at it’s finest. This is where some of my assumptions caught me by surprise. Here is my high-level storage status for the cluster:

Longhorn Storage Dashboard on K3S

At first glance you could make the (reasonable) assumption that:

· 459Gi Schedulable = Storage I Can Use

· 226Gi Reserved = Reserved Storage for Data Parity and Safety

· 68.3Gi Used = Storage that the cluster is actively using

Imagine my surprise then when I carved off 100Gi for a persistent Debian Apt-Cacher container that the volume would fail to mount or do anything meaningful. See, on the surface, everything seems to work. It seems like I would have PLENTY of room for this workload. However, this is where abstraction was lying to me.

To make a long story short, my limitation is NOT the 459Gi number. The REAL limitation is the smallest available schedulable disk on any node that must hold a replica after accounting for:

· Replica count

· Reserved space

· Disk pressure thresholds

This is information that which had not been presented to me on the dashboard. A 100Gi volume with three replicas needs ~300Gi total and each replica MUST fit entirely on a single disk. If one node can’t fit said 100Gi replica…. game over. No volume attachment for you.

Once I put my assumptions aside, got curious, and acknowledged that this was a case where an abstraction layer was hiding the truth from me, I got the manual out and really dug into how Longhorn handles it’s underlying storage. Once I had that deep technical understanding of the underlying storage system, I was able to make informed decisions without solely relying on the abstraction layer, and ultimately get my workload up and off the ground after making different architectural decisions.

It was sobering to run into the very dangers of abstraction that we had JUST gotten done talking about on SysAdmin Weekly so quickly after posting that episode (last week - see below). Also, to be fair to Longhorn, that information is surfaced within the dashboard, just not in THAT particular view. This is, again, how abstraction (and assumptions) can get you into trouble.

So remember, when you run into a situation where things seem one way, but you have that gut feeling that not all appears as it should, ask yourself if you have all the information. Do you truly understand everything under the hood? You’ll be a better SysAdmin for it.

And now…. on to our regularly scheduled programming!

SysAdmin Weekly Announcements

As I’ve mentioned in several episodes of the podcast, we’ve been working on a defined community “hangout” spot that we can use for announcements, community conversations, knowledge sharing, questions, polls…etc…etc. Well I’m happy to announce that said initiative is complete and we now have an official GitHub Discussions repository setup specifically for this purpose.

You can find the SysAdmin Weekly GitHub Discussions Repository Here!

Why GitHub Discussions?

My one hard requirement for any community gathering place was low barrier of entry. Applications like Discord / Slack, while good, require users to download an app and / or setup an account. MOST SysAdmins will already have a GitHub account and GitHub is widely supported in both desktop web browsers as well as on mobile.

The notifications system is simple as well. For example:

· If you just want a quick shortcut to the SysAdmin Weekly discussions repo, just star it.

· If you want granular notifications based on activity, “watch” the repo instead.

What Type of Content Can We Expect There?

You can expect to find:

· Show / Newsletter Announcements

· Discussion Posts for the latest episode / edition of both the podcast AND the newsletter

· Polls about content you’d like to see more of on the show

· Technical Discussions

· Q & A

· Helpful scripts / tools like I posted here

· Friendly Banter

· Etc…

What Do I Need to Do to Get Started?

Outside of setting up a GitHub account in the off chance you don’t have one? Nothing! Feel free to check it out and Introduce yourself, if you’d like! (optional)

🎙 The Latest on the SysAdmin Weekly Podcast

SysAdmin Weekly - 036: The Hidden Dangers of Abstraction in Modern IT

SysAdmin Weekly Podcast Episode 036

Topic: In our first episode of 2026, Andy and Eric dive into the topic of rampant abstraction in modern IT. While yes, abstraction layers have solved many scaling issues over the last decade, there IS a hidden cost. Do SysAdmins still have that deep technical understanding of the entire stack? This episode explores that question in depth!

Why this one matters:

· This episode highlights the dangers of relying heavily on abstracted features & deployments without deep understanding

· Should something break under abstraction layers, SysAdmins are often at the mercy of a vendor to solve the issue.

· This epsiode contains some tips and tricks to help you ensure that your skills are at a point where you can benefit from abstraction layers, but NOT be dependent on them.

Community Signal

Here are some super-cool community links I’ve stumbled upon recently. Highly recommended!

Didier Van Hoye – Do I really need 10 Gbps fiber to the home?

Didier cuts through the marketing noise around ultra-fast residential fiber and lands on a conclusion most practicing SysAdmins already know: raw bandwidth is rarely the real bottleneck. Using his own home setup as an example, he shows how a relatively modest fiber connection is more than sufficient for remote work, lab access, and day-to-day operations, especially when most heavy lifting stays local. The real value here is the reminder that stability, latency, and realistic workload analysis matter far more than chasing headline speeds, and that blindly upgrading capacity without a matching use case is just another form of abstraction-driven decision making.

Luise Freese – Stop using GitHub Copilot as a chatbot!

Luise takes a hard look at the common pattern of treating GitHub Copilot like a chat-based coding partner and explains why that workflow almost always feels fragile and slow. This is not because the model is dumb, but because the interaction model is unstable. Rather than prompt, re-prompt, and steer in chat until something sticks, the article argues that you get far better results by encoding stable constraints and task definitions in files before ever calling Copilot, so the model can actually execute against known invariants instead of chasing shifting context. This post reframes Copilot from a reactive autocomplete toy into a tool that thrives within explicit structure, which is a valuable shift in mindset for anyone trying to use AI in software development.

From AndyOnTech and Project Runspace

AndyOnTech

Regular listeners of the podcast will know I went on something of a…. “rant” during the last podcast episode of the year. In said rant, I explained how there are no longer any good web browsers available in the marketing today. I recently posted a lengthy blog post along the same lines if you’re interested. I’ve linked that below, along with the Podcast episode from late last year on the same topic:

· There Are No Good Web Browsers Left (and It Is a BIG Problem)

· SysAdmin Weekly - 035 - AI Browsers, Chromium Monoculture, and the Future of Browser Security

Handy Tool - Pomocli

Full disclosure: This week I am selfishly promoting my own tool. That said, I can hardly call it a “tool” and more of a “anti-context switcher”. Folks that know me know I’m a BIG markdown guy and love to create my content within the terminal. Context switching is highly impactful on my output, so I go to great lengths to prevent it if possible.

Awhile back I had released a very simple cli tool called Pomocli that acted as a terminal-based pomodoro timer. It would ask what you planned on working on, log that to a text file, then start a timer. I always liked it’s lightness…. but hated how it looked. I recently started tinkering with building basic UIs using Ncurses and Python. As a result, I decided to rebuild Pomocli with a curses-based UI that still fills the same role with the added feature of allowing you to browse previous work sprints.

Pomocli image

You can find Pomocli HERE

I hope you all find it as useful as I do!

Until Next Week

Thanks for reading this week’s edition! I really hope everyone is enjoying the new, more conversational format! If you have any feedback or would like to see a specific topic discussed, be sure to let us know on the SysAdmin Weekly GitHub Discussions Board!

May your systems stay running, and may you avoid maintenance at 5pm on a Friday!

Stay Frosty!

--Andy @ SysAdmin Weekly

· The newsletter is evolving! This means fewer link dumps, more opinion, commentary, and real-world SysAdmin pain from the trenches.

· Theme of the week: Craft vs. Shovelware and why modern software feels more brittle, less thoughtful, and far more hostile to operators.

· A real-world rant with receipts: Microsoft Authenticator quietly falling over once you manage a lot of accounts… and the only fix being “nuke from orbit and start over.”

· The latest SysAdmin Weekly Podcast digs deeper into “good enough” software and why SysAdmins are the ones paying the price for it.

· Community Signal highlights thoughtful takes on whether software is actually getting worse. Plus practical Azure Local update guidance you’ll actually need.

· Other links worth your time this week include one rock-solid security podcast and one non-technical sanity-preserver (because balance matters).

· Looking ahead: plans to spin up GitHub Discussions as a low-friction, SysAdmin-friendly place to keep conversations going beyond the podcast and newsletter.

An Update on the Newsletter

First off, I’m sensitive to the fact that we’ve been away on this newsletter for quite sometime (4 weeks?) and I’m also very aware that I’ve provided MANY “Updates on the Newsletter” like this over the last month or two. The fact of the matter comes down to two things:

1. With the day job, the podcast, and a VERY time intensive side-project I’ve been working on, getting to this newsletter has been inconsistent and difficult.

2. I haven’t been 100% feeling the “vibe” of the newsletter format we’ve been using up until this point. Statistically “Link-dump” newsletter are one of the most common and while, yes, they can provide some value depending on the quality of the links, dumping a collection of links each an every week just to provide a collection of links doesn’t produce CONSISTENT value that I personally find worthy of your time.

Moving forward this newsletter will serve as much more of a commentary and opinion-based newsletter remaining laser-focused on SysAdmins and the challenges they face in the trenches every day. Will there still be some links? Sure, but the format will change to content that is much more conversational in style which provides a better backdrop for me to produce that CONSISTENT value that I want to provide this community.

If you have comments or concerns about the new direction, do let me know in the discussion section below!

From the Console

Our theme for this week’s edition of the newsletter focuses on “Craft vs. Shovelware”. I’ve been in the industry long enough now to recognize that software seems to be more brittle and problematic now than it used to. The amount of times that I’ve had to employ some sort of workaround to a seemingly simple-to-solve bug in order to keep production live has certainly increased over the years. And yes, our environments are wildly more complex than they used to be what with cloud and hybrid deployments and yes, that introduces EXTRA chances for breakage. This does not however account for cases where applications operate in a way that is woefully unhelpful to the user.

I’ll provide my own example on this one. I have been a LONG time user of the Microsoft Authenticator app for MFA and Identity services across a number of tenants in Entra ID and various 3rd party accounts. As of this writing I have around 43 different accounts managed within the Microsoft authenticator app. One day, I tried to add account number 44 to the app and I get hit with a big nasty “device token error”. Long story short, it seems I’ve hit a community-experienced, but non-Microsoft documented issue with the authenticator app where either there is a some soft-limit on number of accounts or there is underlying token corruption. Either way, I am unable to add any more accounts until I rip and restore ALL 43 accounts currently in my authenticator app….. needless to say… I was NOT impressed.

As mentioned, Microsoft has never documented a maximum number of accounts supported by the Microsoft Authenticator app (that I’ve been able to find). However, there is substantial real-world community evidence that the app does not scale reliably for users managing dozens of tenants / accounts. Admins frequently report that once they reach a high number of accounts, Authenticator begins failing to add new accounts or register push notifications, often producing vague registration failures or “device token errors”. One commonly cited message states: “Error changing device token. You must delete and reactivate your accounts to correct the issue,” effectively forcing a full MFA re-registration across all accounts.

Microsoft’s public documentation focuses instead on per-user Entra ID limits (such as the five registered authenticator devices limit), while offering little guidance on app-level scalability or token exhaustion. Community discussions on Microsoft Q&A and Reddit consistently describe these token-related failures and the same recommended remediation of “delete and re-register” suggesting an undocumented Authenticator scalability or push token management issue rather than isolated user misconfiguration.

Which brings me to my point of this whole rant about software quality in the modern day and user-expected app behavior. I’m completely fine with imposed limitations on software (such as max number of supported accounts) as that is something that I, as a SysAdmin, can plan and architect for. What I would have expected in this situation, if my suspicions about scalability and token issues are correct, is a warning at ~30 accounts “Max number of accounts reached”. Instead we get silent breakage, and to quote Ultron from Avengers: Age of Ultron,…“Boom… the end… start again”.

All this is to say, software development needs to once again be given the time and consideration to be a “Craft” again. What we currently seem to have is a hastily built software “content-pipeline” that is designed to push features extra-fast and squeeze maximum revenue from every byte. Something’s got to give.

And now, back to our regularly scheduled programming.

🎙 The Latest on the SysAdmin Weekly Podcast

This week I had regular co-host Eric Siron back on the show to discuss the very issue of “Good Enough” software mentioned above. You’ll find that it’s deeply attuned to this week’s theme.

SysAdmin Weekly Episode 034 Thumbnail Image

Episode: “Good Enough” Software is Ruining IT (and SysAdmins are Paying the Price)

Topic: Software enshitification has been a real issue in the industry lately and there are real consequences (both spoken and not) of that fact. One of those consequences is the fact that SysAdmins are usually the ones that are facing the full brunt of poorly written software that was deemed “good enough” at some point. In this episode Eric and I (Andy) discuss the state of software in our industry while lamenting a time when software was a craft, and NOT just a content pipeline.

Episode links below:

· SysAdmin Weekly on YouTube

· SysAdmin Weekly on Spotify

· SysAdmin Weekly on Apple Podcasts

· SysAdmin Weekly Website

Community Signal

While researching the aforementioned issues regarding this week’s theme. I did happen upon some useful community posts along the same vein or similar:

• This older post on the Stackoverflow Blog by Issac Lyman that asks the question “Is software getting worse?” Issac has a very fair and reasoned look at the question, and even though I’m not a developer, I greatly enjoyed reading it. I highly suggest you check it out.

• I also wanted to include some practical SysAdmin help from the community this week, starting with How to Keep your Azure Local Instances Up to Date by Thomas Maurer! More SysAdmin’s are finding themselves in a situation where they’re starting to rollout Azure Local, so this knowledge is a MUST have.

Other Links Worth Your Time

As I’m a regular podcaster, I often get asked which podcasts do I regularly listen too. I’ll provide one of my favorite technical podcasts, and one more…. hobby related.

• My technical podcast recommendation is very security related, and I’ve mentioned it on SysAdmin Weekly several times. That is the Security Now Podcast featuring Steve Gibson. Steve does an AMAZING job of explaining cybersecurity topics and when you listen to an episode of this show, you’ll learn more about the given topic then you ever thought possible! Highly recommended!

• My NON-technical choice has to do with my long-standing gaming addiction. I’ve been playing video games since I was old enough to hold a controller and I love listening to Remember the Game. Adam (the host) does an amazing job of keeping things interesting and funny. If you’re a long time gamer like me, you won’t regret putting this one on your list.

Building the SysAdmin Weekly Community

A final closing note on my efforts to continue developing our growing community. It’s come to our attention that we need a place that is quicker when it comes to communications. Something that can be used for announcements as well ongoing discussion. While we could certainly comment on each episode posting as well as the newsletter comment sections here, the goal is to have a centralized location where most of the community conversation happens.

As it stands, I’m currently looking at using Github Discussions for this purpose. Most SysAdmins will already have a Github account, and it doesn’t require a new app to be installed, so the barrier to entry is low. Be on the lookout for this new hangout in the coming weeks.

Retro SysAdmin Fact of the Week

Don’t forget. For years, many enterprise systems treated a two-digit year as “good enough” and it could have taken down the world.

The Y2K problem wasn’t a mysterious bug or media hysteria; it was the logical outcome of early storage constraints and yes, short-term thinking. To be fair, saving two bytes per record mattered when memory and disk were measured in kilobytes. Meaning developers stored years as YY instead of YYYY. This “temporary workaround” quietly became the norm. When that calendar rolled to 2000, systems didn’t fail because they were old, they failed because, let’s face it, no one expected them to still be mission-critical 10, 20 or 30 years later. Y2K remains one of the most successful IT and SysAdmin remediation efforts in history.

See? The concept of “we’ll fix it later” has been around for a VERY long time.

Until Next Week

Thanks for reading this week’s edition! If you haven’t yet, be sure to hit that subscribe button. It’s free and you’ll get useful, maximum-uptime quality content in your mailbox every week to help you in the SysAdmin trenches. Also feel free to share with another SysAdmin. For real…. we want to help as many people out of unintended systems outages as possible.

Additionally you can always find all the links associated with this newsletter and the podcast at www.sysadminweekly.com.

Until next week! Stay Frosty!

-Andy
SysAdmin Weekly

· We’ve got a truly spooky Halloween-themed podcast this week (because who doesn’t like horror stories with servers?).

· Major critical patch alert: CVE‑2025‑59287 in Windows Server Update Services (WSUS) is actively exploited - if you run WSUS, patch now.

· We’re revisiting network fundamentals with the OSI model - yes, dust off your layer cake, we’re serving up layer 1 to 7 for your next troubleshooting session.

· Strong community content this week: Linux stress-testing, Azure Local overview, a data-driven look at Apple Silicon CPU core frequencies.

· Tool of the Week: a slick terminal app that could become your new favourite toy.

· Quick Win & Retro Fact - because even infrastructure nerds deserve a little snack.

· Wrap-up with a plug for next week’s podcast (you know you’ll listen).

An Update on the Publication

I greatly appreciate all the readers of this newsletter and you may have noticed an absence of the publication for the edition that was to come out two weeks ago. As folks that watch the podcast know, there has been A LOT of projects (both personal and professional) on my plate over the last month or two. This time of year is typically my busiest time of year and sadly something had to give. That said, I’m through my backlog, and looking forward to continue providing regular editions of this newsletter once again!

This Week’s Insight(s) from Andy

Theme for this edition: “Patch & Pivot: From Fundamental Layers to Active Threats”
In this issue, we’re straddling both ends of the SysAdmin spectrum: going back to the bedrock with the OSI model (don’t roll your eyes yet) and facing forward into the very real, very active world of exploited vulnerabilities (yes, the WSUS one). The juxtaposition is intentional: whether you’re explaining Layer 3 routing to a junior or scrambling to fix a critical RCE in your update infrastructure, the mindset of “understand deep; act fast” is what differentiates the average from the excellent.

When it comes to cybersecurity, and more importantly achieving real security outcomes, the mantra “understand deep; act fast” can feel intimidating for those pivoting into the security space. But here’s the truth: cybersecurity pros are often the ones in the room who understand their tech stack *most* deeply.

Why? Because to truly secure something, you have to understand it from the kernel up. Every layer, every dependency, every quirk. Yet, in a world obsessed with abstraction, we’ve drifted away from the underlying mechanics that keep our systems alive. That blind spot? It’s exactly where threat actors love to play.

So how does that tie into this week’s theme Patch and Pivot? Patching is easy. Anyone can apply an update and move on. But pivoting? That’s where the real challenge lies. SysAdmins are now standing at the crossroads of IT and security, and the ones who thrive will be those who dig deeper the ones who move beyond “apply patch” and into “understand why it matters.” Because real defense starts where surface-level understanding stops.

So if you’re a SysAdmin today, don’t wait to be handed a security title, start learning the internals your tools hide. You can’t protect what you don’t understand.

And now, back to our regularly scheduled programming…..

Latest on the SysAdmin Weekly Podcast

This week’s episode: Halloween “SysAdmin Horror Stories” - join Andy as he and guests spin real-world tales where things went terrifyingly wrong in infrastructure, cloud, and on-prem. Yes, there’s ghosting, haunted servers, and maybe even a spectre of unmanaged firmware creeping through the data centre.

What You Missed on Last Week’s Episode of SysAdmin Weekly

Last week’s episode: University IT training vs real-world readiness - Andy and guest Clay Tamam dove into whether formal IT degrees truly prepare sysadmins for the trenches, or if bootcamps, on-the-job hacks, and real mistakes carry more weight.

Sneak Peek of an Upcoming Episode

Next week: Andy and Eric talk about the “enshitification” of tech vendors and how vendor behavior has shifted from partner to parasite, what happens when the vendor plays you more than you play the tech, and what SysAdmins can do about it.

From AndyOnTech

How to stress-test Linux - A clean, practical walkthrough on putting your Linux boxes through controlled chaos. Covers CPU, memory, GPU, and I/O stress testing with just enough explanation to understand what’s actually happening under the hood. It’s the kind of test you’ll wish you ran *before* production went sideways. Perfect weekend lab project for anyone running a home server or validating hardware stability.

Core Fundamentals

The OSI model explained - It’s old-school, yes, but still gold. Understanding each of the seven layers gives you the lens to pinpoint where things go sideways: from physical cabling (Layer 1) to application behavior (Layer 7).

Helpful Community Content

• How to restore deleted Entra ID conditional access policies and named locations - From Jan Bakker: a lifesaver when someone nukes your conditional access by accident.

• Azure Local overview video - From Thomas Maurer: what Azure Local is, why you might need it, and how it fits with edge/distributed stuff.

• Updated CPU core frequencies for Apple Silicon Macs - Data-driven deep dive for the Apple side of the house. Cool geek reading.

Other SysAdmin Content from Vendors and Official Publications

· Fedora Linux 43 has been released - New features, new kernel, new life for test labs maybe.

· Windows 11 Insider Preview Build 26120.6982 (Beta Channel) announced - If you’re running lab machines or poking at future client OS behavior: heads up.

Security Headlines for SysAdmins

· Critical WSUS vulnerability: CVE-2025-59287 – patch immediately - This one is massive. A remote, unauthenticated RCE in WSUS, CVSS 9.8, active exploits. If you have a WSUS server role enabled: stop reading, go patch.

· Cybersecurity and Infrastructure Security Agency (CISA) updates their Known Exploited Vulnerabilities catalog - They’ve flagged the WSUS vuln among others; this isn’t hypothetical anymore.

Tool of the Week

Ghostty - A slick terminal-application you’ll want to try: for those nights when typing at 3 AM and wondering whether your shell can be cooler. Go ahead, indulge your inner penguin.

Quick Win of the Week

Check your inventory/CMDB and ask the question: “Which servers have the WSUS Server role enabled and are listening on ports 8530 or 8531?” If you find any, prioritize patching or isolating them immediately. You might just win this week before the next alert hits.

Fun Retro SysAdmin Fact

Back in the day, the architecture for the mainframe and minicomputer era relied on multi-user terminals connected via RS-232 lines (often at 9600 bps). That meant many sysadmins were night-shift heroes with endless string lights of terminal screens. Integrity of those lights vs. modern dust-filled racks? It’s evolution, baby.

☕ Wrap-Up

Thanks for riding shotgun with me through this week’s edition of SysAdmin Weekly remember: “Fundamentals + Vigilance = Survival.” If you only take one thing from this newsletter, make it patch early, understand deeply.

Don’t forget to tune into the podcast over at www.sysadminweekly.com and drop us a rating or share your own horror story (especially around midnight). Until next week: stay sharp, stay curious, and yes… stay spooky.