SysAdmin Weekly #19: Trust, But Verify the Abstraction
Why understanding what's under the hood still matters
Welcome back to your regular SysAdmin Weekly Newsletter! After a few weeks off for the holidays plus some time to get our bank of Podcast episodes built back up, here is your first weekly newsletter of 2026!
⏩ TL;DR – This Week in SysAdmin Land
• Abstraction is great… until the dashboard lies to you. A real-world Kubernetes + Longhorn storage lesson on why “schedulable” doesn’t always mean usable.
• Running stateful workloads on Kubernetes still requires understanding the underlying storage mechanics, replicas, disk sizing, and node-level constraints matter.
• SysAdmin Weekly now has an official GitHub Discussions home for announcements, technical discussions, polls, and general sysadmin banter.
• New podcast episode out: The Hidden Dangers of Abstraction in Modern IT that covers why deep technical understanding still matters in 2026.
• Community signal worth your time: a grounded take on why you probably don’t need 10 Gbps home fiber, and a smart rethink of how GitHub Copilot should actually be used.
From the Console
I was reminded to embrace curiosity in IT again this past weekend
In perfect timing with our episode regarding abstraction in modern IT (more on that below), I was reminded to embrace curiosity and not blindly trust / follow abstraction layers in modern software.
As many who watch the podcast know, I’ve been slowly converting many of the workloads in my homelab into containers running on a (now) 4-node K3S (Kubernetes Lite) cluster. This project was brought on by a number of things:
· I’ve always leveraged platforms like Azure Kubernetes Service and wanted to get up to speed on running local Kubernetes
· I’ve got aging hardware in my homelab and getting everything to follow Hyper-V best practices is…. difficult
· Distributed Architecture has always been highly interesting to me
· Many of the workloads I run benefit from being stateless
· I’ve rediscovered my love of all things open-source recently, and have taken steps to start using more stuff from the FOSS ecosystem
While, yes, many of the workloads I run can benefit from being stateless, I have several workloads that MUST be persistent, and as such, require persistent storage for data, state, etc…etc. To facilitate this in a clustered and highly-available fashion, I’ve deployed Longhorn to the cluster to provide distributed block storage across the cluster.
It’s an amazing system really. Unlike hypervisor clusters that require high degrees of compatibility checks, fabric deployment & management, etc, Longhorn makes storage configuration STUPID easy. Once setup and configured on the cluster, anytime a new node is added to the existing cluster, Longhorn will automagically ingest the available storage. Storage is spread in a redundant fashion across all the available nodes and fault domains are automatically defined and configured. Abstraction at it’s finest. This is where some of my assumptions caught me by surprise. Here is my high-level storage status for the cluster:
Longhorn Storage Dashboard on K3S
At first glance you could make the (reasonable) assumption that:
· 459Gi Schedulable = Storage I Can Use
· 226Gi Reserved = Reserved Storage for Data Parity and Safety
· 68.3Gi Used = Storage that the cluster is actively using
Imagine my surprise then when I carved off 100Gi for a persistent Debian Apt-Cacher container that the volume would fail to mount or do anything meaningful. See, on the surface, everything seems to work. It seems like I would have PLENTY of room for this workload. However, this is where abstraction was lying to me.
To make a long story short, my limitation is NOT the 459Gi number. The REAL limitation is the smallest available schedulable disk on any node that must hold a replica after accounting for:
· Replica count
· Reserved space
· Disk pressure thresholds
This is information that which had not been presented to me on the dashboard. A 100Gi volume with three replicas needs ~300Gi total and each replica MUST fit entirely on a single disk. If one node can’t fit said 100Gi replica…. game over. No volume attachment for you.
Once I put my assumptions aside, got curious, and acknowledged that this was a case where an abstraction layer was hiding the truth from me, I got the manual out and really dug into how Longhorn handles it’s underlying storage. Once I had that deep technical understanding of the underlying storage system, I was able to make informed decisions without solely relying on the abstraction layer, and ultimately get my workload up and off the ground after making different architectural decisions.
It was sobering to run into the very dangers of abstraction that we had JUST gotten done talking about on SysAdmin Weekly so quickly after posting that episode (last week - see below). Also, to be fair to Longhorn, that information is surfaced within the dashboard, just not in THAT particular view. This is, again, how abstraction (and assumptions) can get you into trouble.
So remember, when you run into a situation where things seem one way, but you have that gut feeling that not all appears as it should, ask yourself if you have all the information. Do you truly understand everything under the hood? You’ll be a better SysAdmin for it.
And now…. on to our regularly scheduled programming!
SysAdmin Weekly Announcements
As I’ve mentioned in several episodes of the podcast, we’ve been working on a defined community “hangout” spot that we can use for announcements, community conversations, knowledge sharing, questions, polls…etc…etc. Well I’m happy to announce that said initiative is complete and we now have an official GitHub Discussions repository setup specifically for this purpose.
You can find the SysAdmin Weekly GitHub Discussions Repository Here!
Why GitHub Discussions?
My one hard requirement for any community gathering place was low barrier of entry. Applications like Discord / Slack, while good, require users to download an app and / or setup an account. MOST SysAdmins will already have a GitHub account and GitHub is widely supported in both desktop web browsers as well as on mobile.
The notifications system is simple as well. For example:
· If you just want a quick shortcut to the SysAdmin Weekly discussions repo, just star it.
· If you want granular notifications based on activity, “watch” the repo instead.
What Type of Content Can We Expect There?
You can expect to find:
· Show / Newsletter Announcements
· Discussion Posts for the latest episode / edition of both the podcast AND the newsletter
· Polls about content you’d like to see more of on the show
· Technical Discussions
· Q & A
· Helpful scripts / tools like I posted here
· Friendly Banter
· Etc…
What Do I Need to Do to Get Started?
Outside of setting up a GitHub account in the off chance you don’t have one? Nothing! Feel free to check it out and Introduce yourself, if you’d like! (optional)
🎙 The Latest on the SysAdmin Weekly Podcast
SysAdmin Weekly - 036: The Hidden Dangers of Abstraction in Modern IT
SysAdmin Weekly Podcast Episode 036
Topic: In our first episode of 2026, Andy and Eric dive into the topic of rampant abstraction in modern IT. While yes, abstraction layers have solved many scaling issues over the last decade, there IS a hidden cost. Do SysAdmins still have that deep technical understanding of the entire stack? This episode explores that question in depth!
Why this one matters:
· This episode highlights the dangers of relying heavily on abstracted features & deployments without deep understanding
· Should something break under abstraction layers, SysAdmins are often at the mercy of a vendor to solve the issue.
· This epsiode contains some tips and tricks to help you ensure that your skills are at a point where you can benefit from abstraction layers, but NOT be dependent on them.
Community Signal
Here are some super-cool community links I’ve stumbled upon recently. Highly recommended!
Didier Van Hoye – Do I really need 10 Gbps fiber to the home?
Didier cuts through the marketing noise around ultra-fast residential fiber and lands on a conclusion most practicing SysAdmins already know: raw bandwidth is rarely the real bottleneck. Using his own home setup as an example, he shows how a relatively modest fiber connection is more than sufficient for remote work, lab access, and day-to-day operations, especially when most heavy lifting stays local. The real value here is the reminder that stability, latency, and realistic workload analysis matter far more than chasing headline speeds, and that blindly upgrading capacity without a matching use case is just another form of abstraction-driven decision making.
Luise Freese – Stop using GitHub Copilot as a chatbot!
Luise takes a hard look at the common pattern of treating GitHub Copilot like a chat-based coding partner and explains why that workflow almost always feels fragile and slow. This is not because the model is dumb, but because the interaction model is unstable. Rather than prompt, re-prompt, and steer in chat until something sticks, the article argues that you get far better results by encoding stable constraints and task definitions in files before ever calling Copilot, so the model can actually execute against known invariants instead of chasing shifting context. This post reframes Copilot from a reactive autocomplete toy into a tool that thrives within explicit structure, which is a valuable shift in mindset for anyone trying to use AI in software development.
From AndyOnTech and Project Runspace
AndyOnTech
Regular listeners of the podcast will know I went on something of a…. “rant” during the last podcast episode of the year. In said rant, I explained how there are no longer any good web browsers available in the marketing today. I recently posted a lengthy blog post along the same lines if you’re interested. I’ve linked that below, along with the Podcast episode from late last year on the same topic:
· There Are No Good Web Browsers Left (and It Is a BIG Problem)
· SysAdmin Weekly - 035 - AI Browsers, Chromium Monoculture, and the Future of Browser Security
Handy Tool - Pomocli
Full disclosure: This week I am selfishly promoting my own tool. That said, I can hardly call it a “tool” and more of a “anti-context switcher”. Folks that know me know I’m a BIG markdown guy and love to create my content within the terminal. Context switching is highly impactful on my output, so I go to great lengths to prevent it if possible.
Awhile back I had released a very simple cli tool called Pomocli that acted as a terminal-based pomodoro timer. It would ask what you planned on working on, log that to a text file, then start a timer. I always liked it’s lightness…. but hated how it looked. I recently started tinkering with building basic UIs using Ncurses and Python. As a result, I decided to rebuild Pomocli with a curses-based UI that still fills the same role with the added feature of allowing you to browse previous work sprints.
Pomocli image
You can find Pomocli HERE
I hope you all find it as useful as I do!
Until Next Week
Thanks for reading this week’s edition! I really hope everyone is enjoying the new, more conversational format! If you have any feedback or would like to see a specific topic discussed, be sure to let us know on the SysAdmin Weekly GitHub Discussions Board!
May your systems stay running, and may you avoid maintenance at 5pm on a Friday!
Stay Frosty!
--Andy @ SysAdmin Weekly