S3E0: "You Can Tune a System but You Can't Tuna Fish"

Posted 2018-02-26 04:59
Modified 2018-02-26 19:14
Comments 0

In this episode, we talk about our personal, professional, and podcast plans for 2018. We also talk about some basic bird’s-eye-view performance tuning.

As a bonus, you get some Jthan story time, too.

• News
• Notes
• Sysbadministration Award
• Errata
• Music

News

• Spectre and Meltdown
  • We didn’t even bother with sources for this one, because it’s been ALL over the news. The authoritative source, however, is here.
• Details of the result of the Net Neutrality hoopla is here.
• Equifax has lost even more than originally thought.
• The NSA leaked exploits are in the news again.
• Intel told China about the security flaws before the general public.
• A fitness app, Strava, gave away a secret US base location.
• Dutch intelligence supposedly watched Russia hack the DNC and did nothing.
• Intel recommends holding off on patching for now.
  • Intel has also been hit with 3 class action lawsuits relating to the Spectre/Meltdown vulns.
• Amazon refuses to reveal if they hand over Echo data to the US government.
• A journalist ends up essentially leaking her own password hints/passwords.
• Per Nvidia: No more Geforce drivers/GPUs in datacenters. (§ 2.1.3 “Limitations”)

Notes

Starts at 28m37s.

I was drinking Chai tea. Paden was drinking Coors Light. Jthan was drinking water with some lemon in it.

• New year, new us!
  • Personal and professional goals for 2018
    • Jthan: taking more time off work for “me”-time (pottery, “outdoorsy stuff”, etc.), saving money, buying a house and putting down roots. He also wants to focus on his consulting business and getting his RHCSA and RHCE. He wants to work more on direct assistance with workflows/pipelines for clients.
      • He also wants to buy a Honda Ruckus moped. #MakeModGreatAgain
    • r00t^2: I want to move out of the city and settle roots somewhere as well. Professionally, I can’t get into too much detail because NDA but I have a lot of things planned and I’ve been crushing them so far.
    • Paden: Focus on his family. Professionally, pursuing his LPIC cert and homelab.
  • Podcast goals for 2018/S3
    • MERCH. We unanimously agree.
    • More guests.
    • More interaction with our audience.
    • More “review”-type thing. (Speaking of, if you’re a Windows sysadmin and have never used Linux extensively, please contact us!)
  • Jthan found an article that has some good suggestions for the new year for sysadmins.

• Performance tuning (41m4s)
  • Gentoo gives you some MAJOR control over binary compile-time optimizations.
  • We mention it before, but the top 500 supercomputers in the world run GNU/Linux.
  • If you have bare metal and you’re trying to add ONE more service but can’t quite make it, how do you modify/tune what you already have running to make it work? Is this a very bad idea?
    • I propose a normal operating usage should be about 75% resource consumption at average use.
    • Reduce your swappiness! Something to either 0 or 1 should be MUCH more performant.
    • Consider what featuresets/support/libraries are compiled into your binaries.
      • This is why e*Trade used Gentoo.
  • Are there certain distros that lend themselves best to this type of practice? Or OS’s, for that matter?
    • Gentoo is probably the ideal one, here, if you need THAT much control.
  • Service consolidation!
    • Using Apache and Squid or something on the same box? Use Nginx instead! etc.
  • OS/Kernel-level tuning vs. service-level tuning
    • There are a LOT of kernel tunables. Seriously, a lot.
    • We forgot to mention it on the show, but tweaking your preforking handlers (if your services support them) can greatly help with performance on the service end, but can overuse a (small amount) of resources if you’re overly-aggressive with it.
  • Disable kdump (as long as you don’t need to debug kernel panics, etc.)
    • The systemd equivalent I was trying to remember is, appropriately-named, systemd-coredump for userland. You can also still use kdump with it too.
    • Jthan was thinking of abrtd, which is RedHat’s bug reporting daemon.
  • Change your network socket timeouts, etc.
  • Avoid loading extraneous kernel modules and software includes/“drivers” if they aren’t explicitly necessary.
  • Also, run swap from SSD! It’s 2018 and SSD is cheaper than ever. There’s no reason to run swapspace from spinning/platter disks.
  • The main take-away here is, simply, “There is no magic bullet”. You need to really grok the system internals in order to know WHAT to modify, not just HOW to modify them.
    • Some good resources to get started on this journey are:
      • For a very basic introduction, How Linux Works
      • Advanced Programming in the UNIX Environment by W. Richard Stevens (it’s a lot more accessible than the title sounds)
      • The TLDP’s TLK Some explanation on how the Linux kernel works.
      • ESR’s The Art of UNIX Programming
      • And, of course, the ‘Pink Book’

Sysbadministration Award

In this segment, we highlight system administration mistakes. Think of them as the IT equivalent of the Darwin Awards. (57m11s)

Hotspot Shield VPN provider was running a webserver on their customer’s/client-side boxes – and it was vulnerable.

(Shameless self-promotion: if you need a proxy or VPN provider that doesn’t run a webserver on your box or require proprietary software, I gotchu fam.)

Errata

• I had to take over editing for half of the episode; our normal editor had some issues come up. Get better soon, Edita!
  • If any of it sounds shitty, it’s my fault, not hers.
• Also, Jthan was on a different recording rig setup than normal so his audio probably sounds a little different/sub-optimal than normal.
• The duck thing is in S2E22 (the whole duck thing starts at 2h4m30s, continued throughout the rest of the episode, and into the Errata)
• George Michael indeed said “I GOTTA have faith”
  • But if you’re of the younger generation, you probably are more familiar with this cover, lol.
• The images I sent to Jthan for the mod fashion styles are here and here (the first one is the one we talk about on-air).
• Jthan/Paden, Jeep Wranglers do indeed have a higher chance of rolling over – according to SUVs.com, 27.9% (Cherokees aren’t even on the top 5 list).
• You may not be familiar with the concept of the Mandela Effect but you’ve probably gotten the feeling of experiencing it.
• Kia did indeed offer a “Buy one, get one” deal. The advert Jthan mentions is here
• My noodling game is on-point. So is Gordon Ramsay’s.
  • Jthan references the cockulator thing, throwback to S1E22.

Music

Author r00t^2
Categories Season Three