I am trying to create a Minikube image that supports DevOps (via a GitLab cluster agent) and GitOps (via FluxCD). I have some experience with all the tech but combining it all in a small automated package appears to be difficult. I am looking for automating as much as possible.
In order to keep the image small, the minikube start etc. are done once the image is loaded inside a container. A lot is done via a script similar to the following (work in progress!):
#!/bin/bash
echo "Setting up environment"
echo "Activating bash autocomplete"
echo "source /etc/profile.d/bash_completion.sh" >> ~/.bashrc
echo 'Adding alias for "minikube kubectl" as "kubectl"'
echo 'alias kubectl="minikube kubectl"' >> ~/.bashrc
echo "Installing kubectl via minikube and activating bash completion for it"
minikube kubectl completion bash >> ~/.bashrc
echo "Activating bash completion for flux"
flux completion bash > ~/flux_completion.sh
echo "source flux_completion.sh" >> ~/.bashrc
echo "Installing Minikube Kubernetes flavor with Docker driver and starting it"
echo "Adding support for Helm"
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get-helm-3 > get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh
minikube delete && minikube start --force --driver=docker
I am loading and setting everything up via docker-compose:
version: '2'
services:
minikube-otg:
image: minikube-otg:latest
container_name: minikube-otg
restart: always
network_mode: host
volumes:
- /var/run/docker.sock:/var/run/docker.sock
minikube-otg-setup:
image: minikube-otg:latest
depends_on:
- minikube-otg
restart: "no"
command: bash -c "./startup.sh"
If I just enter the container in interactive mode with bash and execute the script myself, it works.
Setting up environment
Activating bash autocomplete
Adding alias for "minikube kubectl" as "kubectl"
Installing kubectl via minikube and activating bash completion for it
Activating bash completion for flux
Installing Minikube Kubernetes flavor with Docker driver and starting it
Adding support for Helm
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11679 100 11679 0 0 173k 0 --:--:-- --:--:-- --:--:-- 175k
Downloading https://get.helm.sh/helm-v3.14.0-linux-amd64.tar.gz
Verifying checksum... Done.
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
! "minikube" profile does not exist, trying anyways.
X Failed to stop ssh-agent process: failed loading config: cluster "minikube" does not exist
* Removed all traces of the "minikube" cluster.
* minikube v1.32.0 on Debian 12.4 (docker/amd64)
! minikube skips various validations when --force is supplied; this may lead to unexpected behavior
* Using the docker driver based on user configuration
* The "docker" driver should not be used with root privileges. If you wish to continue as root, use --force.
* If you are running minikube within a VM, consider using --driver=none:
* https://minikube.sigs.k8s.io/docs/reference/drivers/none/
* Using Docker driver with root privileges
* Starting control plane node minikube in cluster minikube
* Pulling base image ...
* Downloading Kubernetes v1.28.3 preload ...
> preloaded-images-k8s-v18-v1...: 403.35 MiB / 403.35 MiB 100.00% 1.92 Mi
* Creating docker container (CPUs=2, Memory=7900MB) ...
* Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Configuring RBAC rules ...
* Configuring bridge CNI (Container Networking Interface) ...
- Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Verifying Kubernetes components...
* Enabled addons: default-storageclass, storage-provisioner
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
After installing and re-registering (same token) my GitLab agent in a separate step, I get what I expect:
However, the problem is that this setup fails to install Minikube if done via docker-compose. I get the following error:
minikube-otg-setup_1 | ! "minikube" profile does not exist, trying anyways.
minikube-otg-setup_1 | X Failed to stop ssh-agent process: failed loading config: cluster "minikube" does not exist
minikube-otg-setup_1 | * Removed all traces of the "minikube" cluster.
minikube-otg-setup_1 | * minikube v1.32.0 on Debian 12.4 (docker/amd64)
minikube-otg-setup_1 | ! minikube skips various validations when --force is supplied; this may lead to unexpected behavior
minikube-otg-setup_1 | * Using the docker driver based on user configuration
minikube-otg-setup_1 |
minikube-otg-setup_1 | X "docker version --format {{.Server.Os}}-{{.Server.Version}}:{{.Server.Platform.Name}}" exit status 1: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
minikube-otg-setup_1 | * Suggestion: Start the Docker service
minikube-otg-setup_1 | * Documentation: https://minikube.sigs.k8s.io/docs/drivers/docker/
minikube-otg-setup_1 |
minikube-otg-setup_1 |
minikube-otg-setup_1 | X Exiting due to MK_USAGE: Ensure your Docker is running and is healthy.
The error that is the thorn in my foot is
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Howeer, as you can see from my YML configuration the docker.sock should be exposed.
My research shows that it has something to do with specific facilities of container not being up by the time the second service is started (which calls the script to install Minikube). The second service also has a depends_on parameter, but I guess it doesn’t do me any good since Docker does report the first service as healthy and running, hence why the second service starts right away (probably 1 second or so after the first one :D). Then again I might be wrong. Let me know if I can provide more information.