Jonas HietalaModify

Navigation layer

What I did was add a new navigation layer that’s enabled by holding Tab (ring + middle + index on the left-hand side) with all Niri related movement and layout keybinds.

In the graphics above, all green-colored keys emit Gui (which gates all window manager commands) and you can see:

• Long press on Close Window to close a window.

  The long press requirement prevents accidentally closing windows.

• Arrows move through columns/windows.

  Long press resizes them.

• Workspace Up/Down focuses a different workspace.

• Center a column.

• Consume/Expel to combine windows into one column.

  (consume-or-expel-window-left/consume-or-expel-window-right)

• Expand Column makes a column take up all remaining space.

  (expand-column-to-available-width)

• Audio controls.

  To press them I release the index finger (keeping the ring and middle finger pressed to keep the layer active) and use the index to press the audio buttons.

• Mouse buttons.

  In Niri you can move floating windows with Gui + Left Mouse and Gui + Right Mouse to resize them. As my main mouse is a trackball integrated into the keyboard I had to add them to the left-hand side.

  I ended up using QMK’s customizable key repress feature that allows me to:

  1. Press the Tab combo with my three fingers (layer is active)
  2. Release only the index (layer is still active, same as with the audio controls)
  3. Press the index again (now detects the press Gui + Left Mouse key down)
  4. Use the trackball to move the window

  And similarly for the right mouse button to resize with the middle finger. Works great!

Because there are so many commands I want to send I placed Ctrl on the thumb that provides movement-related commands like so:

For example:

• Arrows move columns/windows in the four directions.

• Move columns to the neighboring workspaces.

• Center visible columns.

  (center-visible-columns)

• Slightly different consume/expel semantics.

  (consume-window-into-column/expel-window-from-column)

Regular keymaps

These are triggered in the “normal” way by first pressing the Super combo and then another key on the base layer (I use autoshift so I shift with a long press).

Center the status bar and other Noctalia windows

Feels absolutely required on this screen otherwise things end up in the corners.

Firefox on XWayland

Force Firefox onto XWayland as the Wayland popup manager is broken:

environment {

  MOZ_ENABLE_WAYLAND "0"


}

Dead keys for Ghostty

For some reason dead keys were broken in Ghostty. This is bad for me as the OS keyboard is set to Swedish and it uses them to type ~ (quite a crucial character for a programmer).

The fix:

environment {

  GTK_IM_MODULE "ibus"

  QT_IM_MODULE "ibus"

  XMODIFIERS "@im=ibus"


}

This needs ibus installed and running.

Melange colorscheme

Noctalia discovers custom color schemes under ~/.config/noctalia/colorschemes/<Name>/<Name>.json, so I dropped in my trusty Melange colorscheme there:

{

  "dark": {


    "mPrimary": "#EBC06D",

    "mOnPrimary": "#292522",

    "mSecondary": "#A3A9CE",

    "mOnSecondary": "#292522",

    "mTertiary": "#85B695",

    "mOnTertiary": "#292522",

    "mError": "#D47766",

    "mOnError": "#292522",

    "mSurface": "#292522",

    "mOnSurface": "#ECE1D7",

    "mSurfaceVariant": "#34302C",

    "mOnSurfaceVariant": "#C1A78E",

    "mOutline": "#867462",

    "mShadow": "#1a1816",

    "mHover": "#E49B5D",

    "mOnHover": "#292522",

    "terminal": {


      "normal": {


        "black": "#867462",

        "red": "#D47766",

        "green": "#85B695",

        "yellow": "#EBC06D",

        "blue": "#A3A9CE",

        "magenta": "#CF9BC2",

        "cyan": "#89B3B6",

        "white": "#ECE1D7"


      },

      "bright": {


        "black": "#34302C",

        "red": "#BD8183",

        "green": "#78997A",

        "yellow": "#E49B5D",

        "blue": "#7F91B2",

        "magenta": "#B380B0",

        "cyan": "#7B9695",

        "white": "#C1A78E"


      },

      "foreground": "#ECE1D7",

      "background": "#292522",

      "selectionFg": "#C1A78E",

      "selectionBg": "#403A36",

      "cursorText": "#292522",

      "cursor": "#EBC06D"


    }

  }


}

Then pick the colorscheme:

"colorSchemes": {


    "darkMode": true,

    "predefinedScheme": "Melange",

    "useWallpaperColors": false



}

Layout appearance

The default appearance was pretty I admit but way too much blank space and weirdness. Some tweaks:

layout {

    // Required for noctalia-shell to set wallpaper


    background-color "transparent"



    // Never auto-center focused columns (too much movement)


    center-focused-column "never"

    // But do center a single window


    always-center-single-column



    // No extra space around it all


    struts {}

    // No gaps between windows


    gaps 0

    // The focus ring was annoying


    focus-ring {

        off

    }

    // Use a border with consistent width for all windows instead


    border {

        on

        width 2

        active-color "#ebc06d"

        inactive-color "#403a36"

    }



    // Setting widths is important with such a large screen


    preset-column-widths {

        proportion 0.15

        proportion 0.3

        proportion 0.4

    }

    default-column-width { proportion 0.15; }



    // Heights too, why not?


    preset-window-heights {

        proportion 0.15

        proportion 0.5

        proportion 1.0

    }


}

// Prevent the mouse from opening the overview in the corners


gestures {

    hot-corners {

        off

    }


}

Keep windows centered

Niri has the always-center-single-column option, which is nice as I want to keep as much as possible in the center of the monitor when I’m working. But I very frequently use 2–3 smaller windows and with my frequent opening and closing I’d like them centered too.

Luckily, Niri has an IPC you can use to make a small program that reacts to events and does this for you. I made a small rust project using the niri-ipc crate that does this for me:

[dependencies]


niri-ipc = "26.4.0"

use std::collections::HashMap;


use std::io;




use niri_ipc::socket::Socket;


use niri_ipc::{Action, Event, Request, Response, Window, Workspace};




#[derive(Clone, Copy, PartialEq, Eq, Hash)]


struct WindowId(u64);




#[derive(Clone, Copy, PartialEq, Eq)]


struct WorkspaceId(u64);




struct OutputName<'a>(&'a str);




struct WindowState {


    workspace: Option<WorkspaceId>,

    width: f64,


}




fn main() -> io::Result<()> {


    let mut socket = Socket::connect()?;

    if !matches!(socket.send(Request::EventStream)?, Ok(Response::Handled)) {


        eprintln!("niri rejected event stream");

        std::process::exit(1);

    }



    let mut known: HashMap<WindowId, WindowState> = HashMap::new();

    let mut read_event = socket.read_events();

    loop {


        let result = match read_event()? {


            // A full snapshot of the current state. Just refresh our state.


            Event::WindowsChanged { windows } => {


                known = windows

                    .into_iter()

                    .map(|w| {


                        (


                            WindowId(w.id),

                            WindowState {


                                workspace: w.workspace_id.map(WorkspaceId),

                                width: w.layout.tile_size.0,

                            },

                        )

                    })

                    .collect();

                Ok(())

            }

            Event::WindowOpenedOrChanged { window } => {


                let workspace = window.workspace_id.map(WorkspaceId);

                let entry = WindowState {


                    workspace,

                    width: window.layout.tile_size.0,

                };

                let prev = known.insert(WindowId(window.id), entry);

                match prev {


                    // Don't center floats.


                    _ if window.is_floating => Ok(()),

                    // New window, try to re-center.


                    None => maybe_center_new(&window),

                    // Window changed workspace, try to re-center.


                    Some(state) if state.workspace != workspace => center_focused_if_fits(),

                    // Skip other things.


                    Some(_) => Ok(()),

                }

            }

            Event::WindowClosed { id } => {


                if known.remove(&WindowId(id)).is_some() {


                    center_focused_if_fits()

                } else {


                    Ok(())

                }

            }

            Event::WindowLayoutsChanged { changes } => {


                // Only re-center if the width was changed, otherwise our re-center will


                // loop back indefinitely.


                let mut resized = false;

                for (id, layout) in changes {


                    if let Some(state) = known.get_mut(&WindowId(id)) {


                        if (state.width - layout.tile_size.0).abs() > 0.5 {


                            state.width = layout.tile_size.0;

                            resized = true;

                        }

                    }

                }

                if resized {


                    center_focused_if_fits()

                } else {


                    Ok(())

                }

            }

            _ => Ok(()),

        };



        if let Err(e) = result {


            eprintln!("autocenter: {e}");

        }

    }


}



/// Center a newly created window if the workspace is focused and if there's surrounding free space left.


fn maybe_center_new(window: &Window) -> io::Result<()> {


    let Some(workspace_id) = window.workspace_id.map(WorkspaceId) else {


        return Ok(());

    };

    let Some(focused) = focused_workspace()? else {


        return Ok(());

    };

    if WorkspaceId(focused.id) == workspace_id {


        center_if_fits(&focused)?;

    }

    Ok(())


}



/// Center windows in the focused workspace if there's surrounding free space left.


fn center_focused_if_fits() -> io::Result<()> {


    if let Some(focused) = focused_workspace()? {


        center_if_fits(&focused)?;

    }

    Ok(())


}



/// Center windows in the workspace if there's surrounding free space left.


fn center_if_fits(workspace: &Workspace) -> io::Result<()> {


    let Some(output) = workspace.output.as_deref().map(OutputName) else {


        return Ok(());

    };

    let Some(width) = output_width(output)? else {


        return Ok(());

    };

    if workspace_width(WorkspaceId(workspace.id))? < f64::from(width) {


        center_visible_columns()?;

    }

    Ok(())


}



/// Issue a one-shot query to Niri, wait, and return the response.


fn query(request: Request) -> io::Result<Response> {


    match Socket::connect()?.send(request)? {


        Ok(response) => Ok(response),

        Err(msg) => Err(io::Error::other(msg)),

    }


}



/// Get the focused workspace.


fn focused_workspace() -> io::Result<Option<Workspace>> {


    match query(Request::Workspaces)? {


        Response::Workspaces(ws) => Ok(ws.into_iter().find(|w| w.is_focused)),

        _ => Ok(None),

    }


}



/// Get the width of an output (monitor).


fn output_width(name: OutputName<'_>) -> io::Result<Option<u32>> {


    match query(Request::Outputs)? {


        Response::Outputs(outputs) => Ok(outputs


            .get(name.0)

            .and_then(|o| o.logical.as_ref())

            .map(|l| l.width)),

        _ => Ok(None),

    }


}



/// Calculates the width of all columns in the workspace.


fn workspace_width(workspace_id: WorkspaceId) -> io::Result<f64> {


    let Response::Windows(windows) = query(Request::Windows)? else {


        return Ok(0.0);

    };

    let mut columns: HashMap<usize, f64> = HashMap::new();

    for w in windows {


        if w.workspace_id.map(WorkspaceId) != Some(workspace_id) {


            continue;

        }

        if let Some((col, _)) = w.layout.pos_in_scrolling_layout {


            let width = columns.entry(col).or_insert(0.0);

            *width = width.max(w.layout.tile_size.0);

        }

    }

    Ok(columns.values().sum())


}



/// Send a command to center the visible columns.


fn center_visible_columns() -> io::Result<()> {


    if let Err(msg) = Socket::connect()?.send(Request::Action(Action::CenterVisibleColumns {}))? {


        eprintln!("center-visible-columns rejected: {msg}");

    }

    Ok(())


}

One catch is that if a new window overflows the monitor width, the script won’t center the columns even if there would be free space left afterwards. This is a little weird but it’s consistent with Niri’s center-visible-columns command. I had a small itch to try to hack around it but in the end I left it alone…

Fixed IP

You can use the port forward method above to access ArgoCD but having a fixed IP is nicer, especially if you want to use the argocd CLI (I only used it when trying to get it all working).

I use a separate service for the IP, which allows me to assign an IP outside of the Cilium load balance pool we setup previously (10.1.4.101–10.1.4.255). It looks like this:

apiVersion: v1


kind: Service


metadata:

  name: argocd-server-ip

  namespace: argocd


spec:

  type: ClusterIP

  externalIPs:

    - 10.1.4.51

  selector:

    app.kubernetes.io/name: argocd-server

  ports:

    - name: http

      port: 80

      targetPort: 8080

It simply routes 10.1.4.51:80 to the IP of argocd-server and port 8080. We need to apply it:

kubectl apply -f gitops/bootstrap/argocd.yaml

And you should see it assigned and able to visit it on the web:

$ kubectl get svc -n argocd argocd-server-ip

NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE

argocd-server-ip   ClusterIP   10.105.209.116   10.1.4.51     80/TCP    3d18h

Repository

I use a self-hosted Forgejo instance (hosted in a Proxmox LXC) but you can use whatever you want. You need an API token with read permissions.

ArgoCD documents a declarative setup that we’ll use. For repositories we need to create a secret:

kubectl create secret generic home-ops-repo \


  --namespace argocd \


  --from-literal=type=git \


  --from-literal=url=<repo-url> \


  --from-literal=username=<username> \


  --from-literal=password=<token> \


  --dry-run=client -o yaml \


  | kubeseal --cert infrastructure/sealed-secrets-cert.pem -o yaml \


  > gitops/bootstrap/argocd-repo-secret.yaml

We also need to add the repository label to the generated file so it looks something like this:

apiVersion: bitnami.com/v1alpha1


kind: SealedSecret


metadata:

  name: home-ops-repo

  namespace: argocd


spec:

  encryptedData:

    password: ...

    type: ...

    url: ...

    username: ...

  template:

    metadata:

      name: home-ops-repo

      namespace: argocd

      labels:

        argocd.argoproj.io/secret-type: repository

Apply it:

kubectl apply -f gitops/bootstrap/argocd-repo-secret.yaml

And the repo should show up in the UI or argocd repo list.

Initial admin password

Instead of letting ArgoCD generate a random password we can set the initial password explicitly. It’s possible to do it either via the Helm chart or the declarative way, which I’ll continue with.

After generating it with kubeseal the secret should look like this:

apiVersion: bitnami.com/v1alpha1


kind: SealedSecret


metadata:

  name: argocd-secret

  namespace: argocd


spec:

  encryptedData:

    admin.password: ...

    admin.passwordMtime: ...

    server.secretkey: ...

  template:

    metadata:

      labels:

        app.kubernetes.io/name: argocd-secret

        app.kubernetes.io/part-of: argocd

      name: argocd-secret

      namespace: argocd

With three keys:

• password is a bcrypt hash:

  
  argocd account bcrypt --password 'supersecret'
  

• passwordMtime is the plaintext modification time:

  
  date -u +"%Y-%m-%dT%H:%M:%SZ"
  

• secretkey is a random value:

  
  openssl rand -base64 48
  

  ArgoCD uses server.secretkey to sign session and API tokens, and it will regenerate the key if restarted. I kept getting logged out when I was rebuilding things all over the place and it was annoying so I added this to make it stable.

App-of-apps

root.yaml will be a master “app-of-apps” as it will be an Application that spawns other applications. It looks like this:

apiVersion: argoproj.io/v1alpha1


kind: Application


metadata:

  name: all-apps

  namespace: argocd

  annotations:

    argocd.argoproj.io/sync-wave: "-100"

  finalizers:

    - resources-finalizer.argocd.argoproj.io


spec:

  project: default

  source:

    repoURL: https://git.hietala.xyz/tree/home-ops.git

    targetRevision: master

    path: gitops/bootstrap

  destination:

    server: https://kubernetes.default.svc

    namespace: argocd

  syncPolicy:

    syncOptions:

      - CreateNamespace=true

    automated:

      prune: true

      selfHeal: true

(You’re going to see lots of yaml files like this, better get used to it.)

The important bits are

• The name all-apps in the argocd namespace

• It should be synced before everyone else (sync-wave: -100)

• Should load everything under gitops/bootstrap from my home-ops repo

• I’ll be using pruning and self-healing to tell ArgoCD to keep everything in sync.

This is the entrypoint and it’s the last manual apply command we’ll use, as it’ll load everything else:

kubectl apply -f gitops/root.yaml

Application sets

Inside the bootstrap/ folder lives, among other things, bootstrap/appset.yaml which is responsible for turning subfolders under apps/ into applications and loading them.

This is what I use and it’s an adaption of the code from the aforementioned blog post:

apiVersion: argoproj.io/v1alpha1


kind: ApplicationSet


metadata:

  name: appset

  namespace: argocd


spec:

  # Use Go's engine so we can use basename/path


  goTemplate: true

  # The engine silently fails on missing keys by default


  goTemplateOptions: ["missingkey=error"]

  # Generate from all subfolders from my repo under apps/


  generators:

    - git:

        repoURL: https://git.hietala.xyz/tree/home-ops.git

        revision: master

        directories:

          - path: gitops/apps/*

  template:

    # The Application name will be folder + "-app"


    # Creative, I know.


    metadata:

      name: "{{.path.basename}}-app"

    spec:

      project: default

      # Describes the manifests of the application,


      # which is everything under apps/<app-folder>/


      source:

        repoURL: https://git.hietala.xyz/tree/home-ops.git

        targetRevision: master

        path: "{{.path.path}}"

      # Give the application its own namespace


      destination:

        server: https://kubernetes.default.svc

        namespace: "{{.path.basename}}"

      syncPolicy:

        syncOptions:

          - CreateNamespace=true

          # Needed for larger manifests


          - ServerSideApply=true

          # ArgoCD does a dry run to check for resource kinds


          # it doesn't yet recognize. This breaks some things


          # so turn it off.


          - SkipDryRunOnMissingResource=true

        automated:

          prune: true

          selfHeal: true

I hope the comments make it clear how the above works. The end result is that everything under apps/ will be loaded and organized under its own application.

Manifests under bootstrap/

Even though we manually apply some manifests during the bootstrap sequence, they can still be managed by ArgoCD after the fact. This is solved by simply placing the relevant files in bootstrap/, and they’ll be managed going forward.

These are what we’ve collected so far:

1. Cilium

   We installed Cilium using a helm install command. Here’s a (hopefully) corresponding example of a manifest that ArgoCD can manage:

   
   apiVersion: argoproj.io/v1alpha1
   
   
   kind: Application
   
   
   metadata:
   
     name: cilium
   
     namespace: argocd
   
     annotations:
   
       argocd.argoproj.io/sync-wave: "-1"
   
     finalizers:
   
       - resources-finalizer.argocd.argoproj.io
   
   
   spec:
   
     project: default
   
     source:
   
       repoURL: https://helm.cilium.io
   
       chart: cilium
   
       targetRevision: 1.19.2
   
       helm:
   
         values: |
   
           kubeProxyReplacement: true
   
           k8sServiceHost: 10.1.4.10
   
           k8sServicePort: 6443
   
           l2announcements:
   
             enabled: true
   
           externalIPs:
   
             enabled: true
   
           gatewayAPI:
   
             enabled: true
   
           ipam:
   
             mode: kubernetes
   
           operator:
   
             replicas: 1
   
           securityContext:
   
             privileged: true
   
     destination:
   
       server: https://kubernetes.default.svc
   
       namespace: kube-system
   
     syncPolicy:
   
       syncOptions:
   
         - CreateNamespace=false
   
         - ServerSideApply=true
   
       automated:
   
         prune: true
   
         selfHeal: true
   

   The duplication here is unfortunate but I couldn’t find a way around it.

   At least we can reuse cilium_config.yaml with the load balancer IP specifications that we created during cluster setup. Just plop it under bootstrap/ and we can move on.

2. Gateway API

   Before installing Cilium we also applied the Gateway CRDs using this command:

   
   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml
   

   This is the corresponding YAML manifest (smaller as we don’t pass dozens of options):

   
   apiVersion: argoproj.io/v1alpha1
   
   
   kind: Application
   
   
   metadata:
   
     name: gateway-api
   
     namespace: argocd
   
     annotations:
   
       argocd.argoproj.io/sync-wave: "-2"
   
     finalizers:
   
       - resources-finalizer.argocd.argoproj.io
   
   
   spec:
   
     project: default
   
     source:
   
       repoURL: https://github.com/kubernetes-sigs/gateway-api.git
   
       targetRevision: v1.2.1
   
       path: config/crd/standard
   
     destination:
   
       server: https://kubernetes.default.svc
   
       namespace: kube-system
   
     syncPolicy:
   
       automated:
   
         prune: true
   
         selfHeal: true
   

   Note that I specify sync-wave -2 for the Gateway and -1 for Cilium because the Gateway should sync before Cilium.

3. Sealed Secrets

   We continue in the same manner of converting Helm installations to manifests:

   
   apiVersion: argoproj.io/v1alpha1
   
   
   kind: Application
   
   
   metadata:
   
     name: sealed-secrets
   
     namespace: argocd
   
     annotations:
   
       argocd.argoproj.io/sync-wave: "-1"
   
     finalizers:
   
       - resources-finalizer.argocd.argoproj.io
   
   
   spec:
   
     project: default
   
     source:
   
       repoURL: https://bitnami-labs.github.io/sealed-secrets
   
       chart: sealed-secrets
   
       targetRevision: 2.16.2
   
       helm:
   
         values: |
   
           fullnameOverride: sealed-secrets-controller
   
     destination:
   
       server: https://kubernetes.default.svc
   
       namespace: sealed-secrets
   
     syncPolicy:
   
       syncOptions:
   
         - CreateNamespace=true
   
       automated:
   
         prune: true
   
         selfHeal: true
   

4. ArgoCD

   In addition to the secrets used for the declarative setup (repository, admin password) ArgoCD can also manage itself:

   
   apiVersion: argoproj.io/v1alpha1
   
   
   kind: Application
   
   
   metadata:
   
     name: argocd
   
     namespace: argocd
   
     finalizers:
   
       - resources-finalizer.argocd.argoproj.io
   
   
   spec:
   
     project: default
   
     source:
   
       repoURL: https://argoproj.github.io/argo-helm
   
       chart: argo-cd
   
       targetRevision: 10.1.4
   
       helm:
   
         values: |
   
           configs:
   
             secret:
   
               createSecret: false
   
     destination:
   
       server: https://kubernetes.default.svc
   
       namespace: argocd
   
     syncPolicy:
   
       syncOptions:
   
         - CreateNamespace=false
   
       automated:
   
         prune: true
   
         selfHeal: true
   
   
   ---
   
   # Static IP as shown previously, `---` separates different kinds
   
   
   

Updated bootstrap sequence

This is the updated bootstrap recipe that I manage with Just:

[doc("Bootstrap everything from zero")]


full:

    just bootstrap::cluster

    just bootstrap::cilium

    just secrets::restore_sealed_secrets_private_key

    just bootstrap::sealed_secrets

    just bootstrap::argocd




[doc("Bootstrap ArgoCD")]


[working-directory('../gitops')]


argocd:

    helm install argocd \

        --repo https://argoproj.github.io/argo-helm \

        argo-cd \

        --namespace argocd \

        --create-namespace \

        --version 10.1.4 \

        --set configs.secret.createSecret=false

    # We don't let ArgoCD generate the secret, we supply one:

    kubectl apply -f bootstrap/argocd-secret.yaml

    # Needed to be able to sync from repo.

    kubectl apply -f bootstrap/argocd-repo-secret.yaml

    # Wait for ArgoCD to deploy.

    kubectl rollout status deployment/argocd-server -n argocd --timeout=300s

    # Technically not needed as it'll get loaded on sync, this just speeds it up a little.

    kubectl apply -f bootstrap/argocd.yaml --force-conflicts --server-side

    # Sync everything else from the repo.

    kubectl apply -f root.yaml

The secret

Renovate will pick up RENOVATE_TOKEN in secrets, generated like so:

kubectl create secret generic renovate-token -n renovate \


  --from-literal=RENOVATE_TOKEN='<token>' \


  --dry-run=client -o yaml \


| kubeseal --cert infrastructure/sealed-secrets-cert.pem -o yaml \


  > gitops/apps/renovate/renovate-secret.yaml

We’ll later add the name renovate-token to the main manifest.

The configuration file

Renovate has a bunch of configurations you can add. I’m good with the defaults except to modify the ArgoCD settings to support the custom directory structure I’m using:

{

  "$schema": "https://docs.renovatebot.com/renovate-schema.json",

  "extends": ["config:recommended"],

  "argocd": {


    "managerFilePatterns": ["/gitops/(apps|bootstrap)/.+\\.yaml$/"]


  },

  "kubernetes": {


    "managerFilePatterns": ["/gitops/(apps|bootstrap)/.+\\.yaml$/"]


  }


}

This activates the argocd and kubernetes managers, where argocd watches ArgoCD specific manifests (Application/ApplicationSet) and kubernetes watches plain manifests (such as container images in Deployment).

The manifest

The Application manifest is similar to the ones we’ve seen before:

apiVersion: argoproj.io/v1alpha1


kind: Application


metadata:

  name: renovate

  namespace: argocd

  finalizers:

    - resources-finalizer.argocd.argoproj.io


spec:

  project: default

  source:

    repoURL: https://docs.renovatebot.com/helm-charts

    chart: renovate

    targetRevision: 46.236.3

    helm:

      values: |

        existingSecret: renovate-token

        cronjob:

          schedule: "0 1 * * *"

        renovate:

          config: |

            {

              "platform": "forgejo",

              "endpoint": "https://git.hietala.xyz",

              "repositories": ["tree/home-ops"],

              "gitAuthor": "Renovate Bot <[email protected]>",

              "automerge": false

            }

  destination:

    server: https://kubernetes.default.svc

    namespace: renovate

  syncPolicy:

    syncOptions:

      - CreateNamespace=true

    automated:

      prune: true

      selfHeal: true

Note that we configure the secret (renovate-token), how often it should run, and the repository information.

Commit and push and it should be ready to go. ArgoCD has a really good UI where you can see what’s happening, if you made a mistake somewhere, and also trigger the cron job manually.

Or you can use kubectl, for example:

# See the health status of all applications


kubectl get applications -A


# Trigger the cron job


kubectl create job -n renovate --from=cronjob/renovate renovate-manual


# Watch the logs


kubectl logs -n renovate -l job-name=renovate-manual -f

With this we’ve got our first example of managing a Kubernetes service via ArgoCD. We’ll continue to see more examples of this as the series continues.

A watchface only for me

I like open source but I don’t have any plans of publishing the watchface. The entire point of the watchface is to remove friction in my life, not create another source of anxiety. This way I’m free to change it, rewrite it, pollute it with vibes, and I can focus on making it just good enough for me without having to worry about others.

I encourage you to steal anything you like and create your own watchface; it’s straightforward with Pebble.

Non-uniformity

My brain is very good at filtering away static and uninteresting information. For example, I’ve tried to have Calendar, Habitica, and Todoist widgets on my phone to help me keep track of my events/habits/tasks… But it didn’t take long for my brain to start completely ignoring them despite them taking up 80% of the screen. When I say ignore I mean that literally: I completely stopped noticing them.

So what I wanted to try was to spice it up a little by randomizing the event geometry. See these two examples:

To me the non-uniform prototype is more visually interesting and the changing geometry should hopefully prevent me from filtering them out.

You can argue that the rounder uniform version would look better on the Pebble Round 2, which is fair. Here’s a similar mockup but with the round’s geometry:

I still think the non-uniform variant catches the eye more and solves my specific problem a bit better but there’s no denying that a round layout on a round watch fits very well.

Moving towards 12 o’clock

Another idea I had was instead of statically fixing events to their times I wanted the events to move towards “now” at 12 o’clock. So an event that renders at 3 o’clock is 3 hours away and an event that overlaps 12 is ongoing.

While this prevents me from easily looking back at my day (past events disappear) it makes it easier to feel the urgency as events draw closer. A bit weird perhaps but in practice it feels great.

Here’s the same event through its lifecycle: drifting in from 3 o’clock, ongoing as it crosses 12 (with the countdown band draining in the clock), then gone once it’s in the past.

Something like this:

Completion countdown

This works pretty well but there’s a problem: how to differentiate between an event that will start in one minute and that has been going on for a while? Because the event gets cut off at 12 there’s no way to tell at a glance.

At first I tried to enlarge the event wedge so it touches the clock if it’s ongoing. That was a bit weird so instead I tried to colorize the clock outline with the event that’s ongoing. Then I realized, why not have the outline count down the remaining time of the event, similar to a Time Timer?

In practice this works out wonderfully well.

Starting/stopping timers on a Pebble watchface

This is a post about design, not about implementation, but I still need to address how we can implement a work timer toggle.

The issue is that a Pebble watchface is a background process that cannot interact with Pebble’s physical buttons. You can bind the buttons to quick launch Pebble apps but not watchfaces. A watchface can detect steps and taps on the watch but it cannot communicate with other apps.

There’s a few ways to get around this:

1. Toggle using taps only.

2. Implement it as a Pebble app, not a watchface.

3. Add a new Pebble app (tied to a button) that round trips commands via the companion app back to the watchface.

   
   digraph {
   
   
     rankdir=TB
   
     button [label="Quick launch\nbutton"]
   
     app    [label="Pebble app"]
   
     phone  [label="Companion app\non phone" class="external"]
   
     face   [label="Watchface"]
   
   
   
     button -> app  [label="press"]
   
     app -> phone   [label="toggle"]
   
     phone -> face  [label="toggle"]
   
     { rank=same; app; face }
   
     app -> face [style=invis]
   
   
   }
   

I first tried to implement toggling using taps but it was too unreliable. If I went the app route I’d have to relaunch it all the time, which represents unacceptable friction. What’s left is the janky phone round trip solution.

I already need a companion app for the calendar sync but it still makes me queasy.

How well does it actually work?

It’s a pretty cool setup and it is helping but it still fails quite often.

Most of the time I forget to start the work timer and other times I’m able to add enough steps by waving my hand around like a crazy person, and then continue working. (If I raise the step count then it gets annoying to dismiss when I’m actually walking.)

Could be tuned better I suppose.

Other types of alarms

Visually there are some distinct alarms I use:

• Pause (signals the start of a short break between two working blocks)
• Stop (signals the end of all working blocks)
• End of day (working day completely over)
• Manual alarm (just a normal fire-once alarm without dismissal)

I’ve separated them visually and they each use unique vibrations and tunes.

Date and week number

Here are some prototypes for dates that I liked but ultimately discarded:

The difficulty is to not crowd the watchface too much with the date and the other elements I’ll get to later. In the end I ended up with a simple 31/10 label and a small weekly text without background as I don’t look at the week number that often:

I’m not completely sold on the week number display (yellow labels were a lot more fun) but it’s good enough I guess.

Step counter

It was really difficult for me to find a step counter design I liked. Everything I tried just felt off. Here are three of my attempts:

I used to track steps at the edges for most of the design (you can see the step elements in many of the prototype images in this post) but in the end it introduced a bit too much clutter for my taste. While the other two attempts above are crude and could be polished more I didn’t see potential in them. (Examining and cutting away branches is important during design I think.)

And then the revelation: I don’t care about the exact step count. The only purpose of showing steps is to help me keep the step count up, so I’m not sitting in front of the computer all day, and having badges for tiers solves that problem even better.

Here are the badges I use:

Bronze is achieved at 33% of my daily goal, silver at 66%, gold at 100%, and diamond is for over-achieving with 150%. I feel that having an easy to reach tier helps me do something on the lazy days and the hard-to-reach tier helps give me an extra kick the days I’m out and about.

Yes, not knowing exactly how much is left to the next tier is slightly annoying, but for those cases opening up the Health app on the watch to view the step count is good enough.

Battery states

High after the rush of figuring out how to display steps let’s use the same approach for battery warnings:

Yes, I probably should make the little battery interactive or something but this is a 80:20 situation: I get 80% of the benefit from 20% of the effort. It’s good enough.

Bluetooth & Sleep / DND

More badges!

Pebble has a “Quiet Time” mode that silences notifications. This is a separate thing from Android’s sleep mode or do-not-disturb mode. As I have a companion app I can detect the Android state and the watchface combines the two (and silences task alarms for instance) while also showing a “sleep” bubble so I can see these invisible states.

The next task

I’m a big fan of some of the ideas of Getting Things Done such as the idea of the “next task”: the one actionable thing that you should do next.

I’ve used the Todoist Android widget on my phone to display my tasks but I wanted to incorporate it into my watch too. The idea is to only display the next task, something like this:

While I think all three prototypes above are visually pleasing I went with the wider display that takes up less vertical space:

As a bonus I don’t have to mess around with rendering angled text.

Completing/rescheduling the task

It works fairly well but there’s a wrinkle: when I’ve done the task or realize I can’t do it now I have to pull up my phone and that’s annoying so in practice I’m leaving a stale task alone forever, priming my brain to ignore it.

I’ve tried the Todoist Mini app on the Pebble but there’s so many clicks to find the task and complete or reschedule it.

What I did was add it to the app I’m already using to toggle the work timer and give it a menu of options:

I can still toggle the work timer, complete the next task, or reschedule the task (it’s rescheduled for tomorrow).

I’ve bound it to the UP button via “quick launch”. To keep toggling the work timer as simple as possible pressing UP again toggles the timer (so double clicking UP when the watchface is visible toggles the timer). Otherwise it uses the standard Pebble control scheme (UP/DOWN to select and SEL to execute) and the app closes and returns to the watch when an item is selected.

So far it’s been working really well but as has always been the case before with these productivity “fixes”; I always tire of them eventually.

Edge markers

As a little extra I added some markers to track other types of things along the edges:

The colored dots for calendar events are genuinely useful as many events require some preparation/travel time. In these cases I add a reminder for the event at, say, 15 minutes before. With the dots this preparation time can now also be seen by glancing at the watch (Pebble natively shows reminder alerts anyway.)

The end of day marker is also useful for when I’m planning my working day but sunrise / sunset is mostly a gimmick, and I don’t use the manual alarms .

Just commands

It won’t take long for me to forget these commands so I’ll add them to Just. These recipes will take care of the secrets we’ve handled so far and the cleartext files (talosconfig, kubeconfig, secrets.auto.tfvars, terraform.tfstate) should allow us to regain cluster control, or to bootstrap the cluster anew from the git repo and the sops key.

[doc("Decrypt required files committed to git")]


decrypt_required:

    just secrets decrypt_cluster_config

    just secrets decrypt_terraform_secrets

    just secrets decrypt_terraform_state




[doc("Encrypt talosconfig and kubeconfig")]


[working-directory('../infrastructure')]


encrypt_cluster_config:

    sops --encrypt kubeconfig.yaml > kubeconfig.encrypted.yaml

    sops --encrypt talosconfig.yaml > talosconfig.encrypted.yaml




[doc("Decrypt talosconfig and kubeconfig")]


[working-directory('../infrastructure')]


decrypt_cluster_config:

    sops --decrypt talosconfig.encrypted.yaml > talosconfig.yaml

    sops --decrypt kubeconfig.encrypted.yaml > kubeconfig.yaml

    chmod 600 talosconfig.yaml kubeconfig.yaml




[doc("Encrypt secrets.auto.tfvars")]


[working-directory('../infrastructure')]


encrypt_terraform_secrets:

    sops --encrypt secrets.auto.tfvars > secrets.auto.encrypted.tfvars




[doc("Decrypt secrets.auto.tfvars")]


[working-directory('../infrastructure')]


decrypt_terraform_secrets:

    sops --decrypt secrets.auto.encrypted.tfvars > secrets.auto.tfvars

    chmod 600 secrets.auto.tfvars




[doc("Encrypt terraform.tfstate")]


[working-directory('../infrastructure')]


encrypt_terraform_state:

    sops --encrypt --input-type json --output-type json terraform.tfstate > terraform.encrypted.tfstate




[doc("Decrypt terraform.tfstate")]


[working-directory('../infrastructure')]


decrypt_terraform_state:

    # If the encrypted state doesn't exist yet (fresh repo), skip silently.

    test -f terraform.encrypted.tfstate || exit 0

    sops --decrypt --input-type json --output-type json terraform.encrypted.tfstate > terraform.tfstate

    chmod 600 terraform.tfstate

I also added encryption to create_cluster_config bootstrap command, to make it harder for me to forget to add them to the repo:

[doc("Create talosconfig and kubeconfig")]


[working-directory('../infrastructure')]


create_cluster_config:

    terraform output -raw talosconfig > talosconfig.yaml

    terraform output -raw kubeconfig > kubeconfig.yaml

    just secrets::encrypt_cluster_config

And for the terraform state too (run with just tf::apply instead of a plain terraform apply):

[doc("terraform apply, then re-encrypt state (encrypts even on failure)")]


[working-directory('../infrastructure')]


apply *args:

    #!/usr/bin/env bash

    set -e

    trap 'just secrets::encrypt_terraform_state' EXIT

    terraform apply {{args}}




[doc("terraform destroy, then re-encrypt state (encrypts even on failure)")]


[working-directory('../infrastructure')]


destroy *args:

    #!/usr/bin/env bash

    set -e

    trap 'just secrets::encrypt_terraform_state' EXIT

    terraform destroy {{args}}

Installation

I’ll install the sealed secrets controller using helm:

helm install sealed-secrets \


    --repo https://bitnami-labs.github.io/sealed-secrets \


    sealed-secrets \


    --version 2.16.2 \


    --namespace sealed-secrets \


    --create-namespace \


    --set fullnameOverride=sealed-secrets-controller


# Wait for it to deploy


kubectl rollout status deployment/sealed-secrets-controller -n sealed-secrets

To create secrets on the client we also need the kubeseal command. It wasn’t available on the Void Linux package manager, so let’s do the hard way:

set -x KUBESEAL_VERSION '0.36.1'


curl -OL "https://github.com/bitnami-labs/sealed-secrets/releases/download/v$KUBESEAL_VERSION/kubeseal-$KUBESEAL_VERSION-linux-amd64.tar.gz"


tar -xvzf kubeseal-$KUBESEAL_VERSION-linux-amd64.tar.gz kubeseal


sudo install -m 755 kubeseal /usr/local/bin/kubeseal


rm kubeseal-$KUBESEAL_VERSION-linux-amd64.tar.gz

Create a secret

To create a secret we can either use the cluster (needs an active connection) or offline via a certificate. I prefer the certificate simply because you need to pass fewer arguments (--cert vs --controller-name and --controller-namespace). Here’s how to fetch the certificate:

kubeseal --fetch-cert \


         --controller-name=sealed-secrets-controller \


         --controller-namespace=sealed-secrets \


    > infrastructure/sealed-secrets-cert.pem

(It’s a public key, safe to commit to git.)

And this is how we can use it to generate a secret my-secret with the two fields username and password:

kubectl create secret generic my-secret \


  --namespace some-namespace \


  --from-literal=username="user" \


  --from-literal=password="password1" \


  --dry-run=client -o yaml \


  | kubeseal --cert infrastructure/sealed-secrets-cert.pem \


             --format yaml \


  > gitops/apps/myapp/my-secret.yaml

(There might be other ways to do this. You can generate json files for example, but this works and I don’t care to do research.)

It will be stored in gitops/apps/myapp/my-secret.yaml that we can apply:

kubectl apply -f gitops/apps/myapp/my-secret.yaml

In the future when we get our GitOps setup up the process is the same except we don’t apply the secret; just create, commit, and push and it’ll get applied automatically. It may feel like a lot of effort, but it’s quite nice to work with in day-to-day operations.

The process to update a secret is exactly the same; update the file with new contents and reapply.

View a secret

Check that the secret has been applied:

kubectl get secret my-secret -n some-namespace -o yaml

The data fields username and password shown above will be base64 encoded. Here’s how to print out the password in cleartext:

kubectl get secret my-secret -n some-namespace -o jsonpath='{.data.password}' | base64 -d

Or you can view the secrets in a dashboard such as Headlamp, which is arguably easier.

Surviving a cluster reset

There’s one gotcha to sealed secrets: when the controller is installed it will generate a new public/private key pair so all existing sealed secrets are invalidated. We’d have to reseal all secrets after we reset the cluster, which is highly annoying.

We can circumvent this by exporting the private key, encrypt it with SOPS + Age, and store it in git. Then during the bootstrap process we can import the private key to the controller, allowing it to reuse all existing sealed secrets.

First export the key and encrypt it so we can keep it in git (gitignore sealed-secrets-key.yaml):

kubectl get secret -n sealed-secrets \


  -l sealedsecrets.bitnami.com/sealed-secrets-key=active -o yaml \


     > sealed-secrets-key.yaml


sops --encrypt sealed-secrets-key.yaml > sealed-secrets-key.encrypted.yaml

This needs a .sops.yaml rule:

creation_rules:

  - path_regex: infrastructure/sealed-secrets-key(\.encrypted)?.yaml$

    age: age1rrkgd5yza053qk9m8lp0ww39apdarz7w0rjyq85493g8l9gufgnq9cehzx

Then to import it we simply apply it:

kubectl apply -f sealed-secrets-key.yaml

Just commands

We’ve added a few steps to the bootstrap process to setup the sealed secrets controller:

[doc("Bootstrap everything from zero")]


full:

    just bootstrap::cluster

    just bootstrap::cilium

    just secrets::restore_sealed_secrets_private_key

    just bootstrap::sealed_secrets

[doc("Restore sealed-secrets-key.yaml")]


[working-directory('../infrastructure')]


restore_sealed_secrets_private_key:

    # If the encrypted private key doesn't exist, skip the whole recipe.

    test -f sealed-secrets-key.encrypted.yaml || exit 0

    just secrets::decrypt_sealed_secrets_private_key

    # Don't exit if namespace already exists.

    kubectl create namespace sealed-secrets || true

    # Restore the private key.

    kubectl apply -f sealed-secrets-key.yaml

    rm sealed-secrets-key.yaml

    # We may need to restart the controller, but it may not exist, which is fine.

    kubectl rollout restart deployment/sealed-secrets-controller -n sealed-secrets || true

I’ve tried to safeguard the recipe to not crash if we haven’t created a key or bootstrapped the controller yet.

[doc("Install sealed secrets controller")]


sealed_secrets:

    helm install sealed-secrets \

        --repo https://bitnami-labs.github.io/sealed-secrets \

        sealed-secrets \

        --version 2.16.2 \

        --namespace sealed-secrets \

        --create-namespace \

        --set fullnameOverride=sealed-secrets-controller

    # Wait for it to deploy

    kubectl rollout status deployment/sealed-secrets-controller -n sealed-secrets

And some extra management recipes:

[doc("Fetch sealed-secrets-cert.pem, necessary to encrypt secrets offline")]


[working-directory('../infrastructure')]


fetch_sealed_secrets_cert:

    kubeseal --fetch-cert \

             --controller-name=sealed-secrets-controller \

             --controller-namespace=sealed-secrets \

        > sealed-secrets-cert.pem




[doc("Fetch sealed-secrets-key.yaml")]


[working-directory('../infrastructure')]


fetch_sealed_secrets_private_key:

    kubectl get secret -n sealed-secrets -l sealedsecrets.bitnami.com/sealed-secrets-key=active -o yaml > sealed-secrets-key.yaml

    just secrets::encrypt_sealed_secrets_private_key

    rm sealed-secrets-key.yaml




[doc("Encrypt sealed-secrets-key.yaml")]


[working-directory('../infrastructure')]


encrypt_sealed_secrets_private_key:

    sops --encrypt sealed-secrets-key.yaml > sealed-secrets-key.encrypted.yaml




[doc("Decrypt sealed-secrets-key.yaml")]


[working-directory('../infrastructure')]


decrypt_sealed_secrets_private_key:

    sops --decrypt sealed-secrets-key.encrypted.yaml > sealed-secrets-key.yaml

With this we’re prepared to setup GitOps with ArgoCD in the next post.

Waiting for nodes

Note the dependency wait_for_api above in the line cilium: wait_for_api. This means wait_for_api will run before cilium, which waits until the Kubernetes nodes are ready (albeit not necessarily healthy according to talosctl health):

[private]


wait_for_api:

    until kubectl get nodes 2>/dev/null | grep -q "NotReady\|Ready"; do sleep 5; done

You can call just bootstrap::wait_for_api inside scripts too, which would give you more control over ordering. Dependencies on the other hand are deduplicated and always run before the recipe.

Modules and directory structure

I wanted to organize the recipes a little and group them (just tf::init instead of just tf_init). For that we use modules:

mod argocd 'just/argocd.just'


mod bootstrap 'just/bootstrap.just'


mod secrets 'just/secrets.just'


# etc...




# With this `just` will give you a list of the modules.


[private]


default:

    just -l

Organized like this in the repository:

home-ops/

├── justfile          # `just` entrypoint, loads `just/*`

├── just/

│   ├── bootstrap.just

│   ├── secrets.just

│   ├── cluster.just

│   └── ...

├── infrastructure/   # Terraform, setup in the previous post

└── gitops/           # GitOps using ArgoCD, setup in the future

Working directories

Another nice feature is to be able to set the working directory:

[doc("terraform init")]


[working-directory('../infrastructure')]


init *args:

    terraform init {{args}}

This means I can run just tf::init from anywhere in the entire repo, and it will just work.

Export kubeconfig / talosconfig

export KUBECONFIG := justfile_directory() / "infrastructure/kubeconfig.yaml"


export TALOSCONFIG := justfile_directory() / "infrastructure/talosconfig.yaml"

Every kubectl/talosctl command in any just file now targets the cluster, no matter what directory they’re run from.

Shebang

By default each line runs in its own shell. A #!/usr/bin/env bash at the top converts a recipe into one script, which allows you to define functions, run for loops, or set traps for cleanup.

For example, this recipe runs terraform apply and then encrypts the terraform state even on failure:

[doc("terraform apply, then re-encrypt state (encrypts even on failure)")]


[working-directory('../infrastructure')]


apply *args:

    #!/usr/bin/env bash

    set -e

    trap 'just secrets::encrypt_terraform_state' EXIT

    terraform apply {{args}}

(I encrypt the state so I can safely commit it to git, meaning I can manage terraform from multiple computers, since you need the up-to-date state.)