As a cloud company, we usually build our container cluster on cloud platforms such as GCP or AWS. but sometimes we need to think about how to take to go solution to our customer. So, here comes a challenge - Scale-in our cloud platform and put into pocket (mmm I mean “minikube”)
An example GKE service
Here is a simple example which you (might) build in GKE.
DB: MongoDB with stateful set binding with google persistent disk. Use stateful set from (“Running MongoDB on Kubernetes with StatefulSets”) as an example
Web service(fooBar): a golang application which accesses mongo DB with the load balancer. Because
fooBaris proprietary application which
fooBarimage store in GCR (Google Cloud Registry) not in docker hub.
How to migrate your service from GKE to minikube:
We will just list some note to let you know any tip or note you migrate your service to minikube.
1. Migrate Persistent Volume to minikube.
“Here” is a good article to understand StatefulSet. But however it use some resource you could not use in minikube, such as:
Let me put all yaml setting and give more detail.
apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: mongo spec: serviceName: "mongo" replicas: 3 template: metadata: labels: role: mongo environment: test spec: terminationGracePeriodSeconds: 10 containers: - name: mongo image: mongo command: - mongod - "--replSet" - rs0 - "--smallfiles" - "--noprealloc" ports: - containerPort: 27017 volumeMounts: - name: mongo-persistent-storage mountPath: /data/db - name: mongo-sidecar image: cvallance/mongo-k8s-sidecar env: - name: MONGO_SIDECAR_POD_LABELS value: "role=mongo,environment=test" volumeClaimTemplates: - metadata: name: mongo-persistent-storage annotations: volume.beta.kubernetes.io/storage-class: "fast" spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 100Gi
volumeClaimTemplates provide stable storage using PersistentVolumes provisioned by a PersistentVolume Provisioner. (Use “Google Persistent Disk” on GCP)
The “fast” will specific to use google persistent disk with “SSD” performance. So, in this case our Kubernetes volume claim will specific to google cloud persistent disk.
How we migrate to Google Persistent Disk volume into minikube?
There are two solutions for this:
- Still using volume claim but remove google persistent disk annotation.
In the second case, we could still use
volumeClaimTemplates and just remove volume provider persistent disk
annotation. It will select the best for our system. (for now, it is
volumeClaimTemplates: - metadata: name: minikube-clain spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 10Gi
2. Using GCR in minikube
When you deploy GKE, you usually use GCR (Google Cloud Registry) to storage your docker container. If you want to move your project from GKE to minikube, you have solutions as follows:
- Build your own docker registry. But you will need handle follow things:
--insecure-registryif you use the self-signed key.
- Use public DNS to request CA key (It will against we want to move into minikube)
- Use minikube to connect to GCR directly.
- Minikube has add-on
minikube addons configure registry-creds
- Use temproary token solution (around ~30 mins), here is the reference.
- Minikube has add-on
kubectl delete secret gcr kubectl create secret docker-registry gcr \ --docker-server=https://asia.gcr.io \ --docker-username=oauth2accesstoken \ --docker-password="$(gcloud auth print-access-token)" \ [email protected]
Here is the document from minikube.
Launch your service
Ok, that’s done. If you are wondering how to connect to your web service
fooBar. You can just call
minikube service fooBar.
using busybox to debug
It is very useful when you don’t know what’s happening on your pod, especially for networks storage or network.
cat <<EOF | kubectl create -f - apiVersion: extensions/v1beta1 kind: Deployment metadata: name: busybox spec: selector: matchLabels: app: busybox replicas: 1 template: metadata: labels: app: busybox spec: containers: - name: busybox image: busybox command: - sleep - "86400" EOF kubectl exec -it <BUSYBOX_POD_NAME> -- nslookup redis.default
kubectl exec -it busybox-7fc4f6df6-fhxnp -- nslookup redis.default
Destroy & Cleanup the minikube
Remember to remove the setting file when you uninstall minikube.
minikube delete rm -rf ~/.minikube
Cannot start minikube
It might happen as follow reasons:
- minikube version change.
- docker version change.
- VM has already been destroyed by VirtualBox.
Starting local Kubernetes v1.8.0 cluster... Starting VM... E1124 22:16:44.080926 87887 start.go:150] Error starting host: Temporary Error: Error configuring auth on host: Temporary Error: ssh command error:
rm -rf ~/.minikube