At Payscrypt, we’re working with multiple public chains to support all major cryptocurrencies. In order to provide reliable service, we run all the blockchain full nodes by ourselves in our bare metal Kubernetes cluster.

Here are some techniques we’re using to optimize the performance and to reduce the maintenance cost.

1. Understand the nature of the workload

Blockchain full nodes are typical stateful workloads. They need quite a lot of ram, cpu and disk space. One resource might be overlooked is disk IO. You should be careful if you plan to run Ethereum or EOS full nodes on non-ssd disks. Sharing the same disk by multiple full nodes also might get you into trouble. But sometimes allocating dedicated resource for each full node server might not be economically feasible. So experimenting with different setup and making optimized resource planning will help you building a cluster with better performance.

We’ll use Parity as an example in this post. Here is a minimal yaml file for the StatefulSet resource:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: parity-mainnet
  name: parity-mainnet
spec:
  replicas: 1
  serviceName: parity-mainnet
  selector:
    matchLabels:
      app: parity-mainnet
  template:
    metadata:
      labels:
        app: parity-mainnet
    spec:
      containers:
      - args:
        - --jsonrpc-interface=0.0.0.0
        - --ws-interface=0.0.0.0
        - --pruning=fast
        - --pruning-history=128
        - --tracing=on
        - --cache-size=4096
        - --base-path=/home/parity/.local/share/io.parity.ethereum
        - --port=20303
        - --no-ipc
        image: parity/parity:v2.2.11
        imagePullPolicy: IfNotPresent
        name: parity
        volumeMounts:
        - mountPath: /home/parity/.local/share/io.parity.ethereum
          name: parity-mainnet-data
      nodeSelector:
        kubernetes.io/hostname: server01
      volumes:
      - hostPath:
          path: /data/ssd1/parity-mainnet/share/io.parity.ethereum/docker
          type: DirectoryOrCreate
        name: parity-mainnet-data

2. Use local persistent volume instead of hostpath

This is for people running blockchain full nodes on dedicated servers. In cloud environments like GCP or AWS, we could store all the chain data on external disks provided by the cloud provider and let Kubernetes manage the whole lifecycle of volumes.

For bare metal Kubernetes cluster, one typical approach for persistent storage is to store chain data on Kubernetes nodes using hostpath. It works but it’s not perfect. You have to bind the pod manually to the host using nodeselector if you’re using hostpath. One better way is to use local persistent volume feature introduced in Kubernetes 1.10.

Now let’s add volume to the StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: parity-mainnet
  name: parity-mainnet
spec:
  replicas: 1
  serviceName: parity-mainnet
  selector:
    matchLabels:
      app: parity-mainnet
  template:
    metadata:
      labels:
        app: parity-mainnet
    spec:
      containers:
      - args:
        - --jsonrpc-interface=0.0.0.0
        - --ws-interface=0.0.0.0
        - --pruning=fast
        - --pruning-history=128
        - --tracing=on
        - --cache-size=4096
        - --base-path=/home/parity/.local/share/io.parity.ethereum
        - --port=20303
        - --no-ipc
        image: parity/parity:v2.2.11
        imagePullPolicy: IfNotPresent
        name: parity
        volumeMounts:
        - mountPath: /home/parity/.local/share/io.parity.ethereum
          name: data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "ssd-disks"
      resources:
        requests:
          storage: 200Gi

3. Setup a backup strategy

The data volume keeping all the block data is quite valuable. It took very little efforts to launch a full node but it will take several days or weeks for the full node to download blocks from peers and sync to the latest block.

If the data volume got corrupted or lost, it will took quite a lot of efforts to restore the full node if you don’t have a backup which leads to wasted resource and service outage.

We’re using Stash which use restic to backup the data volume to backblaze every few hours. We’ll cover Stash in another post. Here is the CRD definition we’re using to backup the parity node.

apiVersion: stash.appscode.com/v1alpha1
kind: Restic
metadata:
  name: stash-parity-mainnet
spec:
  selector:
    matchLabels:
      app: parity-mainnet-backup
  type: offline
  fileGroups:
  - path: /source/data
    retentionPolicyName: 'keep-last-30'
  backend:
    b2:
      bucket: YourBucketForParityMainnetBackup
      prefix: backup/parity-mainnet
    storageSecretName: Your-B2-Secret
  schedule: '@every 24h'
  paused: false
  volumeMounts:
  - mountPath: /source/data
    name: data
  retentionPolicies:
  - name: 'keep-last-30'
    keepDaily: 30
    prune: false

4. Monitor

It would be too late to fix a failed server when some end user began to complain about service outage. We need observability for resource usage and notification when something went wrong.

At Payscrypt, We are using Prometheus and Grafana to monitor all the full nodes pods. Despite of the normal resource usage related metrics, we also monitor more metrics using exporters.

Let’s add the Parity exporter as a sidecar to the StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: parity-mainnet
  name: parity-mainnet
spec:
  replicas: 1
  serviceName: parity-mainnet
  selector:
    matchLabels:
      app: parity-mainnet
  template:
    metadata:
      labels:
        app: parity-mainnet
    spec:
      containers:
      - args:
        - --jsonrpc-interface=0.0.0.0
        - --ws-interface=0.0.0.0
        - --pruning=fast
        - --pruning-history=128
        - --tracing=on
        - --cache-size=4096
        - --base-path=/home/parity/.local/share/io.parity.ethereum
        image: parity/parity:v2.2.11
        imagePullPolicy: IfNotPresent
        name: parity
        volumeMounts:
        - mountPath: /home/parity/.local/share/io.parity.ethereum
          name: data
      - image: quay.io/exodusmovement/parity-exporter
        imagePullPolicy: IfNotPresent
        name: parity-exporter
        env:
        - name: PARITY_EXPORTER_LISTEN
          value: '0.0.0.0:8000'
        - name: PARITY_EXPORTER_NODE
          value: 'http://127.0.0.1:8545/'
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "ssd-disks"
      resources:
        requests:
          storage: 200Gi

5. Use pod anti-affinity to schedule full nodes pods

To reduce possible downtime, we would run multiple instances of the same blockchain full nodes in our cluster. Having them running on the same node might cause single point of failure. We could use nodeSelector to bind one pod to one node manually but it’s better to add pod anti-affinity settings which would tell the scheduler to schedule pods into different Kubernetes nodes.

Now the final yaml file looks like this:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: parity-mainnet
  name: parity-mainnet
spec:
  replicas: 1
  serviceName: parity-mainnet
  selector:
    matchLabels:
      app: parity-mainnet
  template:
    metadata:
      labels:
        app: parity-mainnet
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - parity-mainnet
                - parity-mainnet-backup
            topologyKey: kubernetes.io/hostname
      containers:
      - args:
        - --jsonrpc-interface=0.0.0.0
        - --ws-interface=0.0.0.0
        - --pruning=fast
        - --pruning-history=128
        - --tracing=on
        - --cache-size=4096
        - --base-path=/home/parity/.local/share/io.parity.ethereum
        image: parity/parity:v2.2.11
        imagePullPolicy: IfNotPresent
        name: parity
        volumeMounts:
        - mountPath: /home/parity/.local/share/io.parity.ethereum
          name: data
      - image: quay.io/exodusmovement/parity-exporter
        imagePullPolicy: IfNotPresent
        name: parity-exporter
        env:
        - name: PARITY_EXPORTER_LISTEN
          value: '0.0.0.0:8000'
        - name: PARITY_EXPORTER_NODE
          value: 'http://127.0.0.1:8545/'
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "ssd-disks"
      resources:
        requests:
          storage: 200Gi

Conclusion

In this post, we have showed you some best practices to run blockchain full nodes in Kubernetes cluster efficiently. We’re relying on some existing Kubernetes features and tools:

• StatefulSet

• Local Persistent Volume

• Operator

• Pod Anti-Affinity

Running stateful workloads is not as easy as running stateless workloads. By adopting existing Kubernetes features properly, you can run your blockchain full nodes or any stateful servers without hassle.