Run self-managed Kubernetes on Fugoku — kubeadm, k3s, and RKE2 on bare metal or virtual machines

Kubernetes

Run self-managed Kubernetes clusters on Fugoku using the distribution of your choice — kubeadm, k3s, or RKE2. Fugoku provides the underlying bare metal or virtual machines; you bring the Kubernetes control plane and own the entire stack.

Looking for managed Kubernetes? Fugoku's fully managed Kubernetes service (control plane, upgrades, patching) is planned for Q4 2026. The API reference is available at API Reference → Kubernetes. This page covers self-managed Kubernetes you can run today.

Why Run Kubernetes on Fugoku?

Benefit	Description
Bare metal performance	Dedicated CPU, RAM, and NVMe — no noisy neighbors
GPU support	NVIDIA A100 / H100 bare metal plans with full PCIe passthrough
No lock-in	Standard upstream Kubernetes — move anywhere with `kubectl`
Zero egress	Private network traffic between nodes is free
Full control	You own the control plane, kubelet, CNI, and operators
Any CNI	Cilium, Calico, Flannel — your choice
Flexible billing	Hourly, monthly, or reserved bare metal

Distribution Comparison

Distribution	Footprint	Boot Time	Best For
kubeadm	Full upstream K8s	2-3 min	Production clusters, full control, CNCF conformance
k3s	Single binary, ~100 MB	~30s	Edge, dev, single-node, ARM, IoT gateways
RKE2	Full upstream K8s, secure by default	2-3 min	Regulated environments, FIPS, air-gapped, CIS benchmarks

kubeadm

The official CNCF upstream installer. Production-grade, requires more setup, supports any CNI.

k3s

Lightweight distribution by Rancher / SUSE. Single ~100 MB binary, embedded SQLite or etcd, opinionated defaults, ideal for edge and small clusters.

RKE2

Rancher Kubernetes Engine 2 — upstream Kubernetes with security-first defaults. FIPS-compliant, CIS-hardened, suitable for regulated workloads.

Prerequisites

Before you start:

1+ bare metal or VM servers running Ubuntu 22.04/24.04, Rocky 9, or AlmaLinux 9
SSH key access to each server (see SSH Keys)
A private network connecting all nodes
Firewall allowing:
- 6443/tcp — Kubernetes API server (control plane)
- 10250/tcp — kubelet
- 8472/udp — VXLAN (Flannel, default CNI)
- 51820/udp — WireGuard (some Cilium setups)
- 30000-32767/tcp — NodePort services (if used)

Step 1: Provision Servers

Choose a Plan

For a production cluster, use bare metal plans (m4.metal.small or larger) for control plane nodes and either bare metal or VMs for workers. For dev/test, VMs are faster to provision.

Node Type	Recommended Plan	vCPU	RAM	Disk
Control plane	`m4.metal.small`	6	64 GB	2× 960 GB NVMe
Worker (general)	`vm-standard`	4	16 GB	80 GB NVMe
Worker (GPU)	`f4.metal.large-gpu`	24	768 GB	2× 480 GB + 2× 3.8 TB NVMe + 8× A100

Provision via CLI

# Create a private network for cluster traffic
fugoku networks create \
  --name k8s-cluster-net \
  --cidr 10.20.0.0/16 \
  --region ashburn-1

# Provision control plane node
fugoku create server \
  --name k8s-cp-1 \
  --plan m4.metal.small \
  --image ubuntu-24.04 \
  --region ashburn-1 \
  --network k8s-cluster-net \
  --ssh-key laptop

# Provision 3 worker nodes
for i in 1 2 3; do
  fugoku create server \
    --name k8s-worker-$i \
    --plan vm-standard \
    --image ubuntu-24.04 \
    --region ashburn-1 \
    --network k8s-cluster-net \
    --ssh-key laptop
done

Provision via API

# Create control plane server
curl -X POST https://api.fugoku.com/v1/instances \
  -H "X-Fugoku-API-Key: $FUGOKU_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
     "data": {
       "type": "instances",
       "attributes": {
         "name": "k8s-cp-1",
         "providerId": "prov-1",
         "region": "ashburn-1",
         "instanceType": "m4.metal.small",
         "image": "ubuntu-24.04",
         "workerCount": 1
       }
     }
  }'

Allow Cluster Ports in Firewall

# API server
fugoku firewalls add-rule k8s-fw \
  --direction ingress \
  --protocol tcp \
  --port 6443 \
  --source 0.0.0.0/0 \
  --action allow

# kubelet
fugoku firewalls add-rule k8s-fw \
  --direction ingress \
  --protocol tcp \
  --port 10250 \
  --source 10.20.0.0/16 \
  --action allow

# CNI (Flannel VXLAN)
fugoku firewalls add-rule k8s-fw \
  --direction ingress \
  --protocol udp \
  --port 8472 \
  --source 10.20.0.0/16 \
  --action allow

Step 2: Prepare All Nodes

Run on every node (control plane and workers) before installing your distribution.

# Update system
sudo apt update && sudo apt upgrade -y

# Disable swap (required for kubelet)
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

# Load kernel modules
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# sysctl settings required by Kubernetes networking
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system

# Install containerd
sudo apt install -y containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml > /dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd

Step 3a: Install with kubeadm

Install kubeadm, kubelet, kubectl

sudo apt install -y apt-transport-https ca-certificates curl gpg

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | \
  sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /" | \
  sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt update
sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Initialize Control Plane

Run only on the control plane node:

sudo kubeadm init \
  --control-plane-endpoint "k8s-cp-1.fugoku.example:6443" \
  --pod-network-cidr=10.244.0.0/16 \
  --upload-certs

--upload-certs shares control plane certificates so additional control plane nodes can join without manual cert distribution. Capture the kubeadm join output — you'll need it for workers.

Configure kubectl

mkdir -p $HOME/.kube
sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Verify
kubectl get nodes

Join Workers

Run the kubeadm join command printed by kubeadm init on each worker:

sudo kubeadm join k8s-cp-1.fugoku.example:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

Step 3b: Install with k3s

Install k3s Server (Control Plane + Worker)

curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-cidr 10.244.0.0/16 \
  --service-cidr 10.96.0.0/16 \
  --node-external-ip 203.0.113.10 \
  --tls-san k8s-cp-1.fugoku.example

# kubectl is automatically configured
sudo cat /etc/rancher/k3s/k3s.yaml > ~/.kube/config
export KUBECONFIG=~/.kube/config
kubectl get nodes

Join Workers

On the control plane, get the node token:

sudo cat /var/lib/rancher/k3s/server/node-token

On each worker:

curl -sfL https://get.k3s.io | K3S_URL=https://k8s-cp-1.fugoku.example:6443 \
  K3S_TOKEN=<node-token> sh -

k3s uses Traefik and Klipper (a simple local-storage load balancer) by default. To disable these and use your own ingress, see the k3s configuration docs.

Step 3c: Install with RKE2

Install RKE2 Server (Control Plane)

curl -sfL https://get.rke2.io | sudo sh -

sudo systemctl enable --now rke2-server.service

# Wait for node to be ready
sudo /var/lib/rancher/rke2/bin/kubectl get nodes

# Set up kubeconfig
mkdir -p ~/.kube
sudo cp /etc/rancher/rke2/rke2.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
export KUBECONFIG=~/.kube/config

Join Additional Servers

Get the cluster token on the first server:

sudo cat /var/lib/rancher/rke2/server/token

On each additional server:

curl -sfL https://get.rke2.io | sudo sh -
sudo systemctl enable --now rke2-server.service

# Configure to join the cluster
sudo mkdir -p /etc/rancher/rke2
sudo tee /etc/rancher/rke2/config.yaml > /dev/null <<EOF
server: https://k8s-cp-1.fugoku.example:9345
token: <cluster-token>
EOF
sudo systemctl restart rke2-server.service

Join Agents (Workers)

curl -sfL https://get.rke2.io | sudo sh -
sudo systemctl enable --now rke2-agent.service

sudo tee /etc/rancher/rke2/config.yaml > /dev/null <<EOF
server: https://k8s-cp-1.fugoku.example:9345
token: <cluster-token>
EOF
sudo systemctl restart rke2-agent.service

Step 4: Install a CNI

kubeadm and RKE2 do not install a CNI by default — k3s ships with Flannel by default. Choose a CNI based on your needs.

Cilium (Recommended)

Modern eBPF-based CNI with high performance and rich observability.

# Install Helm
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

# Add Cilium repo
helm repo add cilium https://helm.cilium.io/
helm repo update

# Install Cilium
helm install cilium cilium/cilium --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true

Calico

Battle-tested, flexible CNI with strong network policy support.

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

Flannel

Simple, lightweight CNI — the k3s default. Lower feature set but easier to debug.

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

CNI	Performance	Network Policy	Observability	Complexity
Cilium	Highest (eBPF)	L3/L4/L7	Hubble flows	Medium
Calico	High	L3/L4 (L7 via Envoy)	flow logs	Medium
Flannel	Good	None (basic)	Basic	Low

Step 5: Verify the Cluster

kubectl get nodes -o wide
# NAME          STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP
# k8s-cp-1      Ready    control-plane   5m    v1.30.0   10.20.0.10     203.0.113.10
# k8s-worker-1  Ready    <none>          3m    v1.30.0   10.20.0.11     203.0.113.11
# k8s-worker-2  Ready    <none>          3m    v1.30.0   10.20.0.12     203.0.113.12
# k8s-worker-3  Ready    <none>          3m    v1.30.0   10.20.0.13     203.0.113.13

kubectl get pods -A

Deploy a Test Workload

kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=LoadBalancer
kubectl get svc nginx

# Should show EXTERNAL-IP from the load balancer (Fugoku cloud LB or NodePort)

GPU Support

Kubernetes can schedule workloads onto Fugoku GPU bare metal plans using the NVIDIA device plugin.

Prerequisites

Bare metal plan with GPU (e.g., f4.metal.large-gpu, gpu-a100-1)
NVIDIA driver pre-installed on the image, or install via cloud-init

Install NVIDIA Device Plugin

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.16.2/nvidia-device-plugin.yml

Install NVIDIA Driver (if not pre-installed)

Most Fugoku GPU images include the NVIDIA driver. To install manually:

# Ubuntu
sudo apt install -y nvidia-driver-555 nvidia-utils-555

# Verify
nvidia-smi

# Restart kubelet
sudo systemctl restart kubelet

Verify GPU Scheduling

kubectl describe node k8s-gpu-1 | grep -A5 "Capacity"
# capacity:
#   nvidia.com/gpu: 8
#   ...

# Run a GPU test pod
kubectl run gpu-test --rm -it --restart=Never \
  --image=nvidia/cuda:12.4.0-base-ubuntu22.04 \
  --limits=nvidia.com/gpu=1 -- nvidia-smi

GPU Node Labels

kubectl label node k8s-gpu-1 node.kubernetes.io/role=gpu
kubectl label node k8s-gpu-1 hardware-type=NVIDIA-A100

Use these labels with nodeSelector or node affinity to schedule GPU workloads:

apiVersion: v1
kind: Pod
spec:
  nodeSelector:
    hardware-type: NVIDIA-A100
  containers:
  - name: trainer
    image: my-training-image
    resources:
      limits:
        nvidia.com/gpu: 4

Persistent Storage

Fugoku block storage and object storage can back Kubernetes via CSI.

Block Storage CSI

helm repo add fugoku https://charts.fugoku.com
helm install fugoku-csi fugoku/csi-block-storage --namespace kube-system

Storage classes created:

StorageClass	Provisioner	Reclaim
`fugoku-ssd`	block.csi.fugoku.com	Delete
`fugoku-ssd-retain`	block.csi.fugoku.com	Retain

Object Storage CSI (S3-compatible)

helm install fugoku-s3-csi fugoku/csi-object-storage \
  --namespace kube-system \
  --set endpoint=object.fugoku.com

Use in a PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache
spec:
  storageClassName: fugoku-s3
  accessModes: [ReadWriteMany]
  resources:
    requests:
      storage: 1Ti

Ingress

Fugoku Load Balancers can be used as the Kubernetes ingress — assign the LB to a Service of type LoadBalancer, or use an Ingress controller.

Service Type LoadBalancer

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080

Fugoku provisions a load balancer and assigns an Elastic IP. Get the public IP:

kubectl get svc web
# NAME   TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)
# web    LoadBalancer   10.96.10.10    203.0.113.50   80:31234/TCP

Ingress with TLS

# Install cert-manager for Let's Encrypt
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.0/cert-manager.yaml

# Install nginx ingress controller
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace

See Domains & SSL for cert-manager and Let's Encrypt configuration.

Upgrades

kubeadm

# On control plane
sudo apt-mark unhold kubeadm && sudo apt-get update && sudo apt-get install -y kubeadm=1.31.0-1.1 && sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.31.0
sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1 && sudo apt-mark hold kubelet kubectl
sudo systemctl restart kubelet

# On each worker
sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=1.31.0-1.1 kubectl=1.31.0-1.1 && sudo apt-mark hold kubelet kubectl
sudo systemctl restart kubelet

k3s

# Server
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.30.4+k3s1 sh -

# Agent
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.30.4+k3s1 K3S_URL=https://k8s-cp-1.fugoku.example:6443 K3S_TOKEN=<token> sh -

RKE2

sudo systemctl stop rke2-server.service  # or rke2-agent.service
curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION=v1.30.4+rke2r1 sh -
sudo systemctl start rke2-server.service

Backups

Back up etcd (kubeadm/RKE2) or k3s's embedded datastore, plus /etc/kubernetes/ manifests.

kubeadm / RKE2 — etcd snapshot

sudo systemctl stop kubelet

# etcdctl snapshot
sudo ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/peer.crt \
  --key=/etc/kubernetes/pki/etcd/peer.key

sudo systemctl start kubelet

k3s — built-in

sudo k3s etcd-snapshot save
# or with SQLite backend
sudo k3s server --disable-etcd --snapshot-path=/var/lib/rancher/k3s/db

Schedule Automated Snapshots

fugoku snapshots enable \
  --instance k8s-cp-1 \
  --schedule daily \
  --retention 7

Monitoring

# Install metrics-server
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Install kube-prometheus-stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prom prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

The Fugoku platform exposes node metrics in Server detail → Metrics tab, or via fugoku servers stats.

Best Practices

Run ≥3 control plane nodes for HA — odd numbers for etcd quorum
Separate control plane and worker machines — easier upgrades, better blast-radius control
Use a private network for cluster traffic — keep kubelet and CNI traffic off public IPs
Pin API server to a stable DNS name — --control-plane-endpoint makes HA upgrades trivial
Encrypt secrets at rest — enable KMS provider for etcd
Use NetworkPolicy — even on a private network, segment workloads with Cilium or Calico policies
Back up etcd daily — and test restores quarterly
Run cluster API or ArgoCD for declarative cluster management — avoid manual kubectl apply for cluster state
Set resource requests and limits — prevents noisy-neighbor effects
Keep nodes up to date — both OS packages and Kubernetes version
Use GPU node taints — kubectl taint nodes k8s-gpu-1 nvidia.com/gpu=true:NoSchedule and tolerate explicitly

Troubleshooting

Issue	Resolution
Node stuck NotReady	Check kubelet logs (`journalctl -u kubelet`), network connectivity, swap is disabled
kubeadm init fails on preflight	Most common: swap not disabled, containerd not configured, kernel modules not loaded
Pods stuck in ContainerCreating	Check `kubectl describe pod` — usually CNI not ready or image pull failing
CNI pods CrashLoopBackOff	Check pod network CIDR matches `--pod-network-cidr` / `--cluster-cidr`
NodePort service unreachable	Firewall must allow 30000-32767/tcp from source — add firewall rule
API server unreachable from kubectl	Check `--control-plane-endpoint`, DNS resolution, firewall rule for 6443/tcp
etcd quorum loss	Restore from snapshot or recover with `etcdctl snapshot restore`
GPU not schedulable	Check `nvidia-smi` on the node, device plugin pods running, `kubectl describe node` shows `nvidia.com/gpu` capacity
"connection refused" on 10250	kubelet not running or firewall blocking kubelet port
DNS resolution fails cluster-wide	CoreDNS pods failing — check `kubectl logs -n kube-system -l k8s-app=kube-dns`

Getting Help

Documentation: docs.fugoku.com/kubernetes
Community: discord.gg/fugoku
Support: support@fugoku.com
Enterprise K8s: enterprise@fugoku.com

Next Steps:

Provision Instances for cluster nodes
Create a Private Network for cluster traffic
Set Up Load Balancers for ingress
Configure DNS & TLS for public services
Add GPU Nodes for AI workloads

Kubernetes

On this page