What is the ELK Stack?

“ELK” is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch.

Tips

This post reference to the doc. But Pod memory resource limit like:

1
2
3
4
5
6
7
8
9
10
11

......
  - name: ES_JAVA_OPTS
    value: "-Xms2g -Xmx2g"
resources:
  limits:
    cpu: '1'
    memory: 2Gi
  requests:
    cpu: '1'
    memory: 2Gi
......

You have to: resources:limits:memory: 2Gi > value: "-Xms2g -Xmx2g", or OOM.

Deploying

Heketi provides a RESTful management interface which can be used to manage the life cycle of GlusterFS volumes. With Heketi, cloud services like OpenStack Manila, Kubernetes, and Openshift can dynamically provision GlusterFS volumes.

If Heketi is not setup with authentication, then use curl to verify the configuration:

1
2

curl http://<server:port>/hello
Hello from Heketi

You can also verify the configuration using the heketi-cli when authentication is enabled:

1

heketi-cli -server http://<server:port> -user <user> -secret <key> cluster list

Get heketi secret name

1
2
3

$ kubectl get secret
NAME                                 TYPE                                  DATA   AGE
heketi-config-secret                 Opaque                                3      17h

Get heketi secret for Admin access in order to use API as Admin

Configuring GlusterFS

1

$ git clone https://github.com/gluster/gluster-kubernetes.git

Copy the deploy/ directory to the master node of the Kubernetes cluster.

You will have to provide your own topology file. A sample topology file is included in the deploy/ directory (default location that gk-deploy expects) which can be used as the topology for the vagrant libvirt setup. When creating your own topology file:

• Make sure the topology file only lists block devices intended for heketi’s use. heketi needs access to whole block devices (e.g. /dev/sdb, /dev/vdb) which it will partition and format.
• The hostnames array is a bit misleading. manage should be a list of hostnames for the node, but storage should be a list of IP addresses on the node for backend storage communications.

Prepare hosts with storage

Portworx (PX) requires at least some nodes in the cluster to have dedicated storage for Portworx to use. PX will then carve out virtual volumes from these storage pools. In this example, we use a 3.3T block device that exists on each node.

List block devices on worker nodes

1
2
3
4
5
6
7
8

$ lsblk
NAME                              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                                 8:0    0  1.1T  0 disk 
├─sda1                              8:1    0  512M  0 part /boot/efi
└─sda2                              8:2    0  1.1T  0 part 
  ├─w--192--31--16--16--vg-root   253:0    0  1.1T  0 lvm  /
  └─w--192--31--16--16--vg-swap_1 253:1    0  976M  0 lvm  
sdb                                 8:16   0  3.3T  0 disk

Note the storage device sdb, which will be used by PX as one of it’s raw block disks. All the nodes in this setup have the sdb device.