Friday, August 18, 2023

Breaking/fixing my K8S controller

Just a bit of blowing my own horn...

I managed to break the home lab's K8S config while attempting to troubleshoot a friend's cluster, a week or so back. The primary symptom (other than Multus not working) was showing up as a "NoExecute" status for the controller, when listing taints for the nodes. There were also log entries, complaining about not being able to delete sandboxes. This was also causing issues with Falco, which was deploying only 4 of an expected 6 pods (i.e., the DS wasn't installing on the controller), when trying to deploy it with Helm (a story for another time, I think).

In any case, after a number of Google searches and using "kubectl describe" against a few resources, I backtraced it to "Network plugin returns error: cni plugin not initialized". This turned out to be Multus.

Uninstalling and re-installing Multus corrected the issue. K8S then woke up and destroyed the old sandboxes, fired up the missing Falco pods, and the taint on the controller went back to its normal "NoSchedule" status.

Two things learned today:

  1. Piping "kubectl describe ..." into /bin/less is a good troubleshooting tool.
  2. The same YAML file, that you use to install something, can be used to delete it. In other words: "kubectl create -f multus-thick.yaml" for installing and "kubectl delete -f multus-thick.yaml" for uninstalling.

Sunday, August 13, 2023

Prototyping my Falco install

Just spent a couple hours getting Falco + Sidekick + UI + Redis figured out. Following works. Next up: getting it to work in K8s.

#!/bin/bash

docker run -d -p 6379:6379 redislabs/redisearch:2.2.4

docker run -itd --name falco \
           --privileged \
           -v /var/run/docker.sock:/host/var/run/docker.sock \
           -v /proc:/host/proc:ro \
           -e HTTP_OUTPUT_URL=http://192.168.2.22:2801 \
           falcosecurity/falco-no-driver:latest falco --modern-bpf

docker run -itd --name falcosidekick -p 2801:2801 \
           -e WEBUI_URL=http://192.168.2.22:2802 \
           falcosecurity/falcosidekick

docker run -itd --name fs-ui -p 2802:2802 \
           -e FALCOSIDEKICK_UI_REDIS_URL=192.168.2.22:6379 \
           falcosecurity/falcosidekick-ui falcosidekick-ui 


Saturday, July 8, 2023

Krew custom columns

My contribution to the custom-cols plugin for Krew: show what nodes pods are running on.

Create a file ~/.krew/store/custom-cols/v0.0.5/tamplates/node.tpl so that it contains:

 NAME             NODE             STATUS 
 .metadata.name   .spec.nodeName   .status.phase 

The output will look something like:

 tim@cf-desk:~$ kubectl custom-cols -o node pods -n weave 
 NAME                                         NODE   STATUS 
 weave-scope-agent-g9jgh                      cf1    Running 
 weave-scope-agent-gllg5                      cf2    Running 
 weave-scope-agent-kkm2z                      cf3    Running 
 weave-scope-app-658845597b-wnt9b             cf2    Running 
 weave-scope-cluster-agent-84f7b6767c-2vdkw   cf2    Running 

There may also be some value in making it sortable, based on the node. To do so, create another template (I called mine "nodes.tpl")and swap the first and second columns in each row. Then you can pipe the output through the tail and sort commands. Example template:

 NODE              NAME            STATUS 
 .spec.nodeName    .metadata.name  .status.phase 

The output will look something like:

 tim@cf-desk:~$ k custom-cols -o nodes pods -n weave|tail -n +2|sort 
 cf1    weave-scope-agent-g9jgh                      Running 
 cf2    weave-scope-agent-gllg5                      Running 
 cf2    weave-scope-app-658845597b-wnt9b             Running 
 cf2    weave-scope-cluster-agent-84f7b6767c-2vdkw   Running 
 cf3    weave-scope-agent-kkm2z                      Running 

For info: the "-n +2" in the above tells tail to start processing on the second line (i.e., skip the line with the column headers).

Monday, July 18, 2022

Troubleshooting k8s

New command learned today, while a Gitea deployment was stalled in the "ContainerCreating" step. Short version: the following is valuable.

kubectl get events --all-namespaces  --sort-by='.metadata.creationTimestamp'

It's also worthwhile to note that the output from the above is different from the output of:

kubectl get events -A

It turned out that the permissions for a volume were not correct and the PVC mount was timing out.

Tuesday, June 21, 2022

More Vi Tips

Found "Vi Tips for Developers" while jumping around inside the System Administrator's Webring.

Update: this post was flagged (in June 2022) as violating Blogger's content policy (relating to Malware and Viruses). The above content has not been changed. Only this last statement has been added. Please note that the above post does not relate to Malware or Virsues.

Sendmail compiling for the no-server crowd

For anyone who only wants a box to e-mail it's own logs (and not run a server) and that's still trying to figure out how to get the newest version of Sendmail to run without the "Connection refused by 127.0.0.1" error:

   Edit /etc/mail/submit.cf so that the DS line contains the FQDN to your upstream mail server.

   Example: DSmail.myisp.com

You'll also need to set root:smmsp permissions on /var/spool/mqueue.

Hope this saves someone else some time (it took a bit of reading on my part).

Update: this post was flagged (in June 2022) as violating Blogger's content policy (relating to Malware and Viruses). The above content has not been changed. Only this last statement has been added. Please note that the above post does not relate to Malware or Virsues.

Google

Yikes! I fell into this one while cleaning out the spam filters in the comment section. Seems that someone was spamming google1.com. It turns out that that's a legitimate domain, owned by Google. Having it show up in comment spam probably means that it's a test message. The interesting part is if you type "whois google" (with or without the trailing ".com"). You get the following return:
  • GOOGLE.XDNICE.NET
  • GOOGLE.WAIKOOL.COM
  • GOOGLE.TRENDYMP3.NET
  • GOOGLE.TCONV.NET
  • GOOGLE.SKGPUBLISHING.COM
  • GOOGLE.SITNIK.NET
  • GOOGLE.RU286.COM
  • GOOGLE.RU
  • GOOGLE.PAASEI.NET
  • GOOGLE.MOLDOR.COM
  • GOOGLE.MELBOURNEIT.COM.AU
  • GOOGLE.MARS.ORDERBOX-DNS.COM
  • GOOGLE.MADE-IN-NB.COM
  • GOOGLE.IFREEBSD.COM
  • GOOGLE.IE
  • GOOGLE.FUTUREWORKSONLINE.COM
  • GOOGLE.FR
  • GOOGLE.FI
  • GOOGLE.ES
  • GOOGLE.EARTH.ORDERBOX-DNS.COM
  • GOOGLE.DE
  • GOOGLE.CYGRATIS.BE
  • GOOGLE.COM.ZOMBIED.AND.HACKED.BY.WWW.WEB-HACK.COM
  • GOOGLE.COM.VN
  • GOOGLE.COM.UA
  • GOOGLE.COM.SUCKS.FIND.CRACKZ.WITH.SEARCH.GULLI.COM
  • GOOGLE.COM.PLZ.GIVE.A.PR8.TO.AUDIOTRACKER.NET
  • GOOGLE.COM.MX
  • GOOGLE.COM.IS.POWERED.BY.MIKLEFEDOROV.COM
  • GOOGLE.COM.IS.NOT.HOSTED.BY.ACTIVEDOMAINDNS.NET
  • GOOGLE.COM.IS.APPROVED.BY.NUMEA.COM
  • GOOGLE.COM.HAS.LESS.FREE.PORN.IN.ITS.SEARCH.ENGINE.THAN.SECZY.COM
  • GOOGLE.COM.BR
  • GOOGLE.COM.AU
  • GOOGLE.COLORSEE.COM
  • GOOGLE.CO.UK
  • GOOGLE.CO.JP
  • GOOGLE.CNIELIVE.COM
  • GOOGLE.CL
  • GOOGLE.CHENNAIEXPRESS.COM
  • GOOGLE.CH
  • GOOGLE.CANT.SET.THEIR.SERVERS.TO.GENERATE.THE.TRAFFIC.LIKE.CRAWLINGCLOUT.COM
  • GOOGLE.CA
  • GOOGLE.ADRIANP.NET
  • GOOGLE.8LEGS.NET
  • GOOGLE.51-HELP.COM
  • GOOGLE.NET
  • GOOGLE.COM

While some of those are legitimate, many are not. I wonder how much trouble Google has defending their trademark.

Update: this post was flagged (in June 2022) as violating Blogger's content policy (relating to Malware and Viruses). The above content has not been changed. Only this last statement has been added. Please note that the above post does not relate to Malware or Virsues.