|
|
@@ -0,0 +1,371 @@
|
|
|
+# Instalación de cluster RKE2 con Rancher y Longhorn
|
|
|
+
|
|
|
+Las instrucciones salieron de:
|
|
|
+
|
|
|
+[Simple RKE2, Longhorn and Rancher Install](https://ranchergovernment.com/blog/article-simple-rke2-longhorn-and-rancher-install)
|
|
|
+
|
|
|
+## Nombre de host y direcciones del cluster de producción
|
|
|
+
|
|
|
+### PRODUCCIÓN
|
|
|
+
|
|
|
+| Nombre | IP | Cores | Memoria | Disco |
|
|
|
+|--------|----|:-------:|:---------:|:-------:|
|
|
|
+| sramadrke01v | 10.34.7.112 | 4 | 8192 | 200 |
|
|
|
+| sramadrke02v | 10.34.7.113 | 4 | 8192 | 200 |
|
|
|
+| sramadrke03v | 10.34.7.114 | 4 | 8192 | 200 |
|
|
|
+
|
|
|
+## Nombre de host y direcciones del cluster de desarrollo
|
|
|
+
|
|
|
+Se añade un disco adicional a cada nodo para ser usado por rook-ceph
|
|
|
+
|
|
|
+### DEVELOPMENT
|
|
|
+
|
|
|
+| Nombre | IP | Cores | Memoria | Disco | Disco adicional |
|
|
|
+|--------|----|:-------:|:---------:|:-------:|:-------:|
|
|
|
+| sramadrke01v | 10.34.7.118 | 4 | 8192 | 200 | 200 |
|
|
|
+| sramadrke02v | 10.34.7.119 | 4 | 8192 | 200 | 200 |
|
|
|
+| sramadrke03v | 10.34.7.120 | 4 | 8192 | 200 | 200 |
|
|
|
+
|
|
|
+### Configuración de red
|
|
|
+
|
|
|
+ Subred: 10.34.7.0/24
|
|
|
+ Gateway 10.34.7.126
|
|
|
+ Máscara: 255.255.255.128
|
|
|
+ dns1: 10.34.7.42
|
|
|
+ dns2: 10.34.7.88
|
|
|
+ search domain: smoes.int
|
|
|
+ vlan 3: 10.34.7.x (servidores)
|
|
|
+ vlan 2: dhcp
|
|
|
+ Proxy http://139.16.225.135:81
|
|
|
+
|
|
|
+## Máquina de trabajo.
|
|
|
+
|
|
|
+Instalar Ubuntu Desktop.
|
|
|
+
|
|
|
+NOTA: A partir de aquí hay que ejecutar los comandos como root, hasta que se diga lo contrario.
|
|
|
+
|
|
|
+## Proxy para apt (para actualizar sistema e instalar cosas)
|
|
|
+
|
|
|
+Crear el fichero
|
|
|
+
|
|
|
+ /etc/apt/apt.conf.d/95proxy.conf
|
|
|
+
|
|
|
+y añadir la siguiente línea:
|
|
|
+
|
|
|
+ Acquire::http::Proxy "http://139.16.225.135:81/";
|
|
|
+ Acquire::https::Proxy "http://139.16.225.135:81/";
|
|
|
+
|
|
|
+### Desactivar firewall
|
|
|
+
|
|
|
+
|
|
|
+ # Ubuntu instructions
|
|
|
+ # stop the software firewall
|
|
|
+ systemctl stop ufw
|
|
|
+ systemctl disable ufw
|
|
|
+
|
|
|
+ # get updates, install nfs, and apply
|
|
|
+ apt update
|
|
|
+ apt install nfs-common -y
|
|
|
+ apt upgrade -y
|
|
|
+
|
|
|
+ # clean up
|
|
|
+ apt autoremove -y
|
|
|
+
|
|
|
+
|
|
|
+## Instalación de ubuntu server
|
|
|
+
|
|
|
+Instalar un ubuntu-server en 3 de las máquinas.
|
|
|
+
|
|
|
+Proxies:
|
|
|
+
|
|
|
+ HTTP_PROXY=http://139.16.225.135:81/
|
|
|
+ HTTPS_PROXY=http://139.16.225.135:81/
|
|
|
+ http_proxy=http://139.16.225.135:81/
|
|
|
+ https_proxy=http://139.16.225.135:81/
|
|
|
+ no_proxy=localhost,127.0.0.0/8,::1,<direcciones de los nodos>
|
|
|
+ NO_PROXY=localhost,127.0.0.0/8,::1,<direcciones de los nodos>
|
|
|
+
|
|
|
+Hay que sustituir **\<direcciones de los nodos\>** por las direcciones descritas en la tabla de arriba.
|
|
|
+
|
|
|
+Esto hay que ponerlo en el fichero (para curl las que valen son las mayúsculas)
|
|
|
+
|
|
|
+ /etc/environment
|
|
|
+
|
|
|
+### Instalar RKE2
|
|
|
+
|
|
|
+#### Proxy
|
|
|
+
|
|
|
+las variables de entorno de los Proxy tienen que estar puestas en la shell, en concreto:
|
|
|
+
|
|
|
+ HTTP_PROXY=http://139.16.225.135:81/
|
|
|
+ HTTPS_PROXY=http://139.16.225.135:81/
|
|
|
+ NO_PROXY=localhost,127.0.0.0/8,::1
|
|
|
+
|
|
|
+de lo contrario el
|
|
|
+
|
|
|
+ systemctl start rke2-server.service
|
|
|
+
|
|
|
+da error
|
|
|
+
|
|
|
+#### RKE2 Server
|
|
|
+
|
|
|
+ curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -
|
|
|
+
|
|
|
+##### Proxy para que arranque.
|
|
|
+
|
|
|
+en el fichero
|
|
|
+
|
|
|
+ /etc/default/rke2-server
|
|
|
+
|
|
|
+o, para el agente en
|
|
|
+
|
|
|
+ /etc/default/rke2-agent
|
|
|
+
|
|
|
+añadir
|
|
|
+
|
|
|
+ HTTP_PROXY=http://139.16.225.135:81/
|
|
|
+ HTTPS_PROXY=http://139.16.225.135:81/
|
|
|
+ NO_PROXY=localhost,127.0.0.1,<direcciones del master y de los workers>
|
|
|
+
|
|
|
+si se usa el ```private.rancher.lab``` registry, hay que añadir a ```NO_PROXY``` la dirección del servidor donde esté corriendo. En el caso que nos ocupa, está en ```sramadrke01v``` (10.34.7.112). También creo que vale poner ```*.rancher.lab```.
|
|
|
+
|
|
|
+##### Lanzar rke2
|
|
|
+
|
|
|
+ # start and enable for restarts -
|
|
|
+ systemctl enable rke2-server.service
|
|
|
+ systemctl start rke2-server.service
|
|
|
+
|
|
|
+### Post instalación.
|
|
|
+
|
|
|
+ # simlink all the things - kubectl
|
|
|
+ ln -s $(find /var/lib/rancher/rke2/data/ -name kubectl) /usr/local/bin/kubectl
|
|
|
+
|
|
|
+A partir de aquí, cambiar a usuario normal (no root)
|
|
|
+
|
|
|
+#### Copiar configuración a directorio de usuario
|
|
|
+
|
|
|
+
|
|
|
+ mkdir $HOME/.kube/
|
|
|
+ sudo cp /etc/rancher/rke2/rke2.yaml $HOME/.kube/config
|
|
|
+ sudo chown $(id -un):$(id -gn) .kube/config
|
|
|
+
|
|
|
+ # check node status
|
|
|
+ kubectl get node
|
|
|
+
|
|
|
+¡ojo! el fichero rke2.yaml tiene permisos solo para root. Yo lo he copiado al fichero
|
|
|
+
|
|
|
+ $HOME/.kube/config
|
|
|
+
|
|
|
+y le he puesto como propietario al usuario con el que trabajo.
|
|
|
+
|
|
|
+Este fichero se puede copiar en la máquia de gestión también y así ejecutar los comandos kubectl desde allí. En este caso, hay que modificar el fichero para cambiar la ip del control-plane.
|
|
|
+
|
|
|
+Además, es necesario copiar el fichero
|
|
|
+
|
|
|
+ /var/lib/rancher/rke2/server/node-token
|
|
|
+
|
|
|
+que necesitaremos para que se conecten los agentes, una vez instalados.
|
|
|
+
|
|
|
+#### RKE2 agent
|
|
|
+
|
|
|
+Todo lo relativo al proxy es aplicable aquí.
|
|
|
+
|
|
|
+We add ```INSTALL_RKE2_TYPE=agent```
|
|
|
+
|
|
|
+ curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=agent sh -
|
|
|
+
|
|
|
+Create config file
|
|
|
+
|
|
|
+ mkdir -p /etc/rancher/rke2/
|
|
|
+
|
|
|
+Change the ip to reflect your rancher1 ip
|
|
|
+
|
|
|
+ vi /etc/rancher/rke2/config.yaml
|
|
|
+
|
|
|
+añadir
|
|
|
+
|
|
|
+ server: https://$RANCHER1_IP:9345
|
|
|
+
|
|
|
+al final del fichero.
|
|
|
+
|
|
|
+change the Token to the one from rancher1
|
|
|
+
|
|
|
+ /var/lib/rancher/rke2/server/node-token
|
|
|
+
|
|
|
+añadir
|
|
|
+
|
|
|
+ token: $TOKEN
|
|
|
+
|
|
|
+al final del fichero
|
|
|
+
|
|
|
+##### Lanzar rke2
|
|
|
+
|
|
|
+ # enable and start
|
|
|
+
|
|
|
+ systemctl enable rke2-agent.service
|
|
|
+ systemctl start rke2-agent.service
|
|
|
+
|
|
|
+## RANCHER via Helm
|
|
|
+
|
|
|
+Los comandos helm de instalación y repositorios de los charts hay que hacerlo con cuenta de usuario, no como **root**
|
|
|
+
|
|
|
+
|
|
|
+ # on the server rancher1
|
|
|
+ # add helm
|
|
|
+ curl -#L https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
|
|
|
+
|
|
|
+ # add needed helm charts
|
|
|
+ helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
|
|
|
+ helm repo add jetstack https://charts.jetstack.io
|
|
|
+
|
|
|
+
|
|
|
+ # still on rancher1
|
|
|
+ # add the cert-manager CRD
|
|
|
+ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.crds.yaml
|
|
|
+
|
|
|
+ # helm install jetstack
|
|
|
+ helm upgrade -i cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace
|
|
|
+
|
|
|
+ # helm install rancher
|
|
|
+ helm upgrade -i rancher rancher-latest/rancher --create-namespace --namespace cattle-system --set hostname=<CAMBIAME> --set bootstrapPassword=bootStrapAllTheThings --set replicas=1
|
|
|
+
|
|
|
+## OPCIÓN: Longhorn
|
|
|
+
|
|
|
+
|
|
|
+ # get charts
|
|
|
+ helm repo add longhorn https://charts.longhorn.io
|
|
|
+
|
|
|
+ # update
|
|
|
+ helm repo update
|
|
|
+
|
|
|
+ # install
|
|
|
+ helm upgrade -i longhorn longhorn/longhorn --namespace longhorn-system --create-namespace
|
|
|
+
|
|
|
+### Exponer el servicio
|
|
|
+
|
|
|
+Para poder acceder al UI de longhorn, hay que exponer el servicio. En nuestro caso será mediante LoadBalancer:
|
|
|
+
|
|
|
+ kubectl patch service longhorn-frontend -n longhorn-system -p '{"spec": {"type": "LoadBalancer"}}'
|
|
|
+
|
|
|
+### longhorn - configuración multipath
|
|
|
+
|
|
|
+Esto es importante porque, de lo contrario, longhorn empezará a fallar porque no puede montar los PV
|
|
|
+
|
|
|
+Esta configuración hay que aplicarla en cada una de las máquinas donde corra ```longhorn```, en principio, en todos y cada uno de los nodos.
|
|
|
+
|
|
|
+To prevent multipath daemon from adding additional block devices created by Longhorn
|
|
|
+
|
|
|
+First check devices created by Longhorn using lsblk
|
|
|
+
|
|
|
+ root@localhost:~# lsblk
|
|
|
+ NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
|
|
|
+ sda 8:0 0 79.5G 0 disk /
|
|
|
+ sdb 8:16 0 512M 0 disk [SWAP]
|
|
|
+ sdc 8:32 0 1G 0 disk /var/lib/kubelet/pods/c2c2b848-1f40-4727-8a52-03a74f9c76b9/volumes/kubernetes.io~csi/pvc-859bc3c9-faa8-4f54-85e4-b12935b5ae3c/mount
|
|
|
+ sdd 8:48 0 1G 0 disk /var/lib/kubelet/pods/063a181a-66ac-4644-8268-9215305f9b73/volumes/kubernetes.io~csi/pvc-837eb6ac-45fe-4de7-9c97-8d371eb02190/mount
|
|
|
+ sde 8:64 0 1G 0 disk /var/lib/kubelet/pods/4c80842d-7257-4b91-b668-bb5b111da003/volumes/kubernetes.io~csi/pvc-c01cee3e-f292-4979-b183-6546d6397fbd/mount
|
|
|
+ sdf 8:80 0 1G 0 disk /var/lib/kubelet/pods/052dadd9-042a-451c-9bb1-2d9418f0381f/volumes/kubernetes.io~csi/pvc-ba7a5c9a-d84d-4cb0-959d-3db39f34d81b/mount
|
|
|
+ sdg 8:96 0 1G 0 disk /var/lib/kubelet/pods/7399b073-c262-4963-8c7f-9e481272ea36/volumes/kubernetes.io~csi/pvc-2b122b42-141a-4181-b8fd-ce3cf91f6a64/mount
|
|
|
+ sdh 8:112 0 1G 0 disk /var/lib/kubelet/pods/a63d919d-201b-4eb1-9d84-6440926211a9/volumes/kubernetes.io~csi/pvc-b7731785-8364-42a8-9e7d-7516801ab7e0/mount
|
|
|
+ sdi 8:128 0 1G 0 disk /var/lib/kubelet/pods/3e056ee4-bab4-4230-9054-ab214bdf711f/volumes/kubernetes.io~csi/pvc-89d37a02-8480-4317-b0f1-f17b2a886d1d/mount
|
|
|
+ root@localhost:~#
|
|
|
+
|
|
|
+Notice that Longhorn device names start with ```/dev/sd[x]```
|
|
|
+
|
|
|
+Create the default configuration file ```/etc/multipath.conf``` if not existed
|
|
|
+
|
|
|
+Add the following line to blacklist section devnode "^sd[a-z0-9]+"
|
|
|
+
|
|
|
+ blacklist {
|
|
|
+ devnode "^sd[a-z0-9]+"
|
|
|
+ }
|
|
|
+
|
|
|
+Restart multipath service
|
|
|
+
|
|
|
+ # systemctl restart multipathd.service
|
|
|
+
|
|
|
+
|
|
|
+Verify that configuration is applied
|
|
|
+
|
|
|
+ # multipath -t
|
|
|
+
|
|
|
+## OPCIÓN: rook-ceph
|
|
|
+
|
|
|
+### Desplegar el operador
|
|
|
+
|
|
|
+Desde el directorio ```rook```
|
|
|
+
|
|
|
+ kubectl create -f crds.yaml -f common.yaml -f operator.yaml
|
|
|
+
|
|
|
+### Crear el clúster
|
|
|
+
|
|
|
+ kubectl create -f cluster.yaml
|
|
|
+
|
|
|
+### Comprobar funcionamiento
|
|
|
+
|
|
|
+la salida del comando:
|
|
|
+
|
|
|
+ kubectl -n rook-ceph get pods
|
|
|
+
|
|
|
+tiene que ser así:
|
|
|
+
|
|
|
+ NAME READY STATUS RESTARTS AGE
|
|
|
+ csi-cephfsplugin-4qmbt 3/3 Running 0 21h
|
|
|
+ csi-cephfsplugin-hvg5l 3/3 Running 0 21h
|
|
|
+ csi-cephfsplugin-provisioner-7c7c6bf59-ffhdt 6/6 Running 0 21h
|
|
|
+ csi-cephfsplugin-provisioner-7c7c6bf59-hnt9h 6/6 Running 0 21h
|
|
|
+ csi-cephfsplugin-qp2xg 3/3 Running 0 21h
|
|
|
+ csi-rbdplugin-d7bjx 3/3 Running 0 21h
|
|
|
+ csi-rbdplugin-mxw52 3/3 Running 0 21h
|
|
|
+ csi-rbdplugin-provisioner-654995cf9-ftjr9 6/6 Running 0 21h
|
|
|
+ csi-rbdplugin-provisioner-654995cf9-txd2n 6/6 Running 0 21h
|
|
|
+ csi-rbdplugin-xjq72 3/3 Running 0 21h
|
|
|
+ rook-ceph-crashcollector-sramadrke01v-dev-54479f7864-8jp7r 1/1 Running 0 21h
|
|
|
+ rook-ceph-crashcollector-sramadrke02v-dev-58db65444c-9z6tc 1/1 Running 0 21h
|
|
|
+ rook-ceph-crashcollector-sramadrke03v-dev-7658cb4c8b-qf2xn 1/1 Running 0 21h
|
|
|
+ rook-ceph-exporter-sramadrke01v-dev-67dd8675f-2f5b6 1/1 Running 0 21h
|
|
|
+ rook-ceph-exporter-sramadrke02v-dev-758555d548-57d9s 1/1 Running 0 21h
|
|
|
+ rook-ceph-exporter-sramadrke03v-dev-7fb4f57c4f-b6qnh 1/1 Running 0 21h
|
|
|
+ rook-ceph-mgr-a-7b88459667-5s55f 3/3 Running 0 21h
|
|
|
+ rook-ceph-mgr-b-b99c7cd76-bbgnv 3/3 Running 0 21h
|
|
|
+ rook-ceph-mon-a-7d7cc9b64f-bcm7s 2/2 Running 0 21h
|
|
|
+ rook-ceph-mon-b-794ddfb8c9-jfgvv 2/2 Running 0 21h
|
|
|
+ rook-ceph-mon-c-57648c8d95-pd7dv 2/2 Running 0 21h
|
|
|
+ rook-ceph-operator-84c99b765c-rw7tt 1/1 Running 0 21h
|
|
|
+ rook-ceph-osd-0-564f9fbb76-nnvpr 2/2 Running 0 21h
|
|
|
+ rook-ceph-osd-1-84cd94d6cf-wjsvj 2/2 Running 0 21h
|
|
|
+ rook-ceph-osd-2-7d6676974d-9n6c7 2/2 Running 0 20h
|
|
|
+ rook-ceph-osd-prepare-sramadrke01v-dev-w884k 0/1 Completed 0 21h
|
|
|
+ rook-ceph-osd-prepare-sramadrke02v-dev-n2znr 0/1 Completed 0 21h
|
|
|
+ rook-ceph-osd-prepare-sramadrke03v-dev-fdcfc 0/1 Completed 0 21h
|
|
|
+ rook-ceph-tools-9dfb9b69d-rz7pr 1/1 Running 0 6h18m
|
|
|
+
|
|
|
+Tardará un poco. Bastante si está detrás del proxy.
|
|
|
+
|
|
|
+### Ceph toolbox
|
|
|
+
|
|
|
+La ```rook toolbox``` se instala con
|
|
|
+
|
|
|
+ kubectl create -f toolbox.yaml
|
|
|
+
|
|
|
+una vez que el estado del deployment sea ```running``` se puede entrar para comprobar si está funcionando
|
|
|
+
|
|
|
+ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
|
|
|
+
|
|
|
+con
|
|
|
+
|
|
|
+ ceph status
|
|
|
+
|
|
|
+se puede ver el estado.
|
|
|
+
|
|
|
+### Crear la ```storageclass```
|
|
|
+
|
|
|
+ kubectl create -f storageclass.yaml
|
|
|
+
|
|
|
+Desde el interfaz de Rancher, se puede poner esta ```storageclass``` como por defecto para que las pvc se creen directamente ahí (en el caso de no especificar ninguna)
|
|
|
+
|
|
|
+# Registry privado
|
|
|
+
|
|
|
+Hay que montar un registro privado para nuestras imágenes.
|
|
|
+
|
|
|
+Las instrucciones están [aquí](https://code.siemens.com/celestino.rey/private-docker-registry.git)
|