Use Docker and K8s

Here we use the lightweight K3s as an example. If you have already deployed a K8s cluster, you can refer to the subsequent configuration for similar settings.

DANGER

Please read the entire tutorial before proceeding, otherwise the deployment result may not meet expectations.

Install Docker

You can find the installation instructions on the Docker official website, and use the basic configuration in Quick Start to run GZCTF.

Install K3s

K3s is a lightweight k8s distribution that can be quickly deployed on single and multiple machines. Official documentation: https://docs.k3s.io/

If you only have one machine?

If you only have one machine, we strongly recommend using other methods for deployment:

  • For private intranet competitions with less than 200 teams and 30 challenges, you can directly use Docker for deployment without using k8s.
  • For public or larger competitions, it is strongly recommended to use k3s for integrated deployment, do not use Docker.

Finally, if you really want to run k3s through Docker, you can specify the Docker backend by adding the following parameters during installation, we strongly do NOT recommend you to do this, you may need to fix various compatibility issues and cause a lot of trouble.

INSTALL_K3S_EXEC="--docker"
INFO

If you need to run more than 255 challenge containers on a single k3s instance, you need to specify INSTALL_K3S_EXEC during k3s installation and change node-cidr-mask-size to the desired subnet size.

INSTALL_K3S_EXEC="--kube-controller-manager-arg=node-cidr-mask-size=16"

The above configuration CANNOT BE CHANGED after installation. Changing the CIDR to /16 can support 65535 Pods.

The above configuration only changes the IP address range used by the node Pods. If you need to change the node's Pod limit, please refer to the following.

And install k3s in the following way. For more information, please refer to k3s installation configuration:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="..." sh -
INFO

Chinese users can use a mirror site to speed up installation.

curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -

For multi-node installation and cluster setup, please refer to the official documentation.

Configure GZCTF

The connection configuration file for k3s is located at /etc/rancher/k3s/k3s.yaml, and it can be exported using the following command:

sudo cat /etc/rancher/k3s/k3s.yaml

Use the following command to obtain the IP address of the k3s control-panel machine:

sudo k3s kubectl cluster-info
INFO

If it shows 127.0.0.1, it means that the k3s control-panel is on the current machine. Please use ip a to check the IP address of the current machine.

You can directly use the IP address or use a domain name, but make sure the domain name resolves to the machine where the k3s control-panel is located, and ensure that the machine running GZCTF can access port 6443 on it.

Save the above output as kube-config.yaml and change the server field to the IP address of the machine where the k3s control-panel is located, for example:

apiVersion: v1
clusters:
  - cluster:
      certificate-authority-data: # ...
      server: https://127.0.0.1:6443 # change this to your k3s control-panel's IP or domain
    name: default
# ...

Save it to the machine where GZCTF is deployed, in the same folder as compose.yml, for example, kube-config.yaml. Then modify the mount configuration in compose.yml:

gzctf:
  image: gztime/gzctf:latest
  restart: always
  ports:
    - "80:8080"
  networks:
    default:
  volumes:
    - "./data/files:/app/files"
    - "./appsettings.json:/app/appsettings.json:ro"
    - "./kube-config.yaml:/app/kube-config.yaml:ro" # this is required for k8s deployment
    # - "/var/run/docker.sock:/var/run/docker.sock" # this is required for docker deployment
  depends_on:
    - db

Also, change the appsettings.json file and set the ContainerProvider field:

{
  "Type": "Kubernetes",
  "PublicEntry": "ctf.example.com" // change this to your k3s control-panel's IP or domain
}

Restart GZCTF, and then you can use k3s as the container backend. Users who have already used k8s can also refer to the above configuration process to integrate GZCTF into an existing k8s cluster.

Change NodePort Port Range

The default NodePort port range for k3s is 30000-32767, which may not meet your requirements. Therefore, you can modify the NodePort port range of k3s according to your needs.

Run the following commands on the machine where the k3s control-panel is located:

  • sudo nano /etc/systemd/system/k3s.service

  • Edit the ExecStart setting below to specify service-node-port-range

    ExecStart=/usr/local/bin/k3s \
        server \
        --kube-apiserver-arg=service-node-port-range=20000-50000
  • sudo systemctl daemon-reload

  • sudo systemctl restart k3s

Change the container limit of K3s

The default container limit of K3s is 110, which may not be suitable for a large number of small containers in a competition. Therefore, you can change the container limit of K3s according to your needs.

Run the following commands on the machine where the k3s control-panel is located:

  • sudo nano /etc/rancher/k3s/kubelet.config

  • Edit the maxPods setting as follows

    apiVersion: kubelet.config.k8s.io/v1beta1
    kind: KubeletConfiguration
    maxPods: 800
  • sudo nano /etc/systemd/system/k3s.service

  • Edit the ExecStart setting below to specify kubelet-arg

    ExecStart=/usr/local/bin/k3s \
        server \
        --kubelet-arg=config=/etc/rancher/k3s/kubelet.config
  • sudo systemctl daemon-reload

  • sudo systemctl restart k3s

Add Container Image Registry

Using an external container image registry directly is not supported in k3s. You need to add the image registry to k3s.

Run the following commands on the machine where the k3s control-panel is located:

  • sudo nano /etc/rancher/k3s/registries.yaml

  • Edit the mirrors setting below to specify the address of the image registry you need

    mirrors:
      "container.ctf.example.com": # change this to your registry's domain
        endpoint:
          - "https://container.ctf.example.com" # change this to your registry's domain
  • sudo systemctl restart k3s