CrowdStrike Falcon LogScale on Kubernetes

November 10, 2022

In this how-to guide, we’ll use Kubernetes to deploy a CrowdStrike Falcon® LogScale cluster on our local machine. We’ll walk through the installation of all of the prerequisite tools, the Humio operator, and then create a LogScale cluster. At each stage, we will verify that our components are up and running.

Note that our local installation of this distributed system has multiple components; in production, this would require considerable resources. This means that your local machine will need a decent amount of available CPU and memory to accommodate all the requirements. For actual sizing information and requirements, please visit this page.

Also, deploying LogScale on-premises requires a license. If you’re new to using LogScale, previously known as Humio, you can sign up for a trial license here. Licenses typically require two business days to be issued.

Finally, the content below is for demonstration purposes only and shouldn’t be used in production. Alright, let’s get to it.

Install the Prerequisite Tools

In order to prepare our local environment to run LogScale, we’ll need to install the following tools:

Let’s walk through these one at a time.

Docker

We’ll create our local Kubernetes cluster using kind, which stands for “Kubernetes in Docker.” So, we’ll need to install Docker, too.

Depending on your local machine, you have different options for a Docker solution, including:

Rancher Desktop, which bundles in the installation of kubectl and Helm. Follow the installation instructions here.
Colima, if you are using Mac OS. Follow the installation instructions here.

Before moving on, let’s verify that Docker is installed and operational. Use the docker version command to ensure you see a Client section and a Server section.

$ docker version
Client:
Version: 20.10.9
API version: 1.41
Go version: go1.16.8
Git commit: c2ea9bc
Built: Thu Nov 18 21:17:06 2021
OS/Arch: darwin/arm64
Context: colima
Experimental: true

Server:
Engine:
Version: 20.10.11
API version: 1.41 (minimum version 1.12)
Go version: go1.16.4
Git commit: 847da184ad5048b27f5bdf9d53d070f731b43180
Built: Fri Nov 19 03:41:34 2021
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: v1.5.8
GitCommit: 1e5ef943eb76627a6d3b6de8cd1ef6537f393a71
runc:
Version: 1.0.0-rc95
GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
docker-init:
Version: 0.19.0
GitCommit:

It is important to configure your VM with enough resources. We recommend at least 4 CPU cores and 16 GB of memory.

kind

Kind is an official local Kubernetes cluster. It was designed for testing Kubernetes itself but is very useful for local development and testing. Follow the installation instructions here.

Next, we verify that kind was installed correctly:

$ kind version
kind v0.12.0 go1.17.8 darwin/arm64

kubectl

Kubectl is the official CLI of Kubernetes. You can interact with any Kubernetes cluster from the command-line using kubectl (as long as you have the proper certificate). Unless your Docker installation included kubectl, you can follow the installation instructions here.

Let’s verify that kubectl was installed correctly:

$ kubectl version
Client Version: version.Info{Major:”1″, Minor:”23″, GitVersion:”v1.23.4″, GitCommit:”e6c093d87ea4cbb530a7b2ae91e54c0842d8308a”, GitTreeState:”clean”, BuildDate:”2022-02-16T12:38:05Z”, GoVersion:”go1.17.7″, Compiler:”gc”, Platform:”darwin/amd64″}
Server Version: version.Info{Major:”1″, Minor:”23″, GitVersion:”v1.23.4″, GitCommit:”e6c093d87ea4cbb530a7b2ae91e54c0842d8308a”, GitTreeState:”clean”, BuildDate:”2022-03-06T21:39:59Z”, GoVersion:”go1.17.7″, Compiler:”gc”, Platform:”linux/arm64″}

If you see just a Client version, that’s not a problem. It means your kubectl is not configured to talk to any Kubernetes cluster yet. The output would look like this:

$ kubectl version
I0615 18:48:57.753042 14099 versioner.go:58] invalid configuration: no configuration has been provided
Client Version: version.Info{Major:”1″, Minor:”23″, GitVersion:”v1.23.4″, GitCommit:”e6c093d87ea4cbb530a7b2ae91e54c0842d8308a”, GitTreeState:”clean”, BuildDate:”2022-02-16T12:38:05Z”, GoVersion:”go1.17.7″, Compiler:”gc”, Platform:”darwin/amd64″}
The connection to the server localhost:8080 was refused – did you specify the right host or port?

Helm

Helm is the mainstream Kubernetes package manager. Follow the installation instructions here. Let’s confirm we have Helm 3 running (Helm 2 is obsolete):

$ helm version
version.BuildInfo{Version:”v3.8.2″, GitCommit:”6e3701edea09e5d55a8ca2aae03a68917630e91b”, GitTreeState:”clean”, GoVersion:”go1.17.5″}

Create a kind Cluster

With all of the prerequisites in place, let’s create our kind cluster. This step takes approximately 30 seconds.

$ kind create cluster –name logscale
Creating cluster “logscale” …
✓ Ensuring node image (kindest/node:v1.23.4) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to “kind-logscale”
You can now use your cluster with:

kubectl cluster-info –context kind-logscale

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Let’s verify our cluster is up and running.

$ kubectl cluster-info –context kind-logscale
Kubernetes control plane is running at https://127.0.0.1:65000
CoreDNS is running at https://127.0.0.1:65000/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump’.

Install Kafka using the Strimzi operator

Apache Kafka is required by LogScale. One of the easiest ways to install Kafka is via the Strimzi operator. Let’s install it.

First, we create a kafka namespace.

$ kubectl create ns kafka
namespace/kafka created

Then, we apply the Strimzi YAML manifest:

$ kubectl create \

-f ‘https://strimzi.io/install/latest?namespace=kafka’ \

-n kafka

customresourcedefinition.apiextensions.k8s.io/strimzipodsets.core.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkas.kafka.strimzi.io created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced created
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-broker created
customresourcedefinition.apiextensions.k8s.io/kafkaconnects.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkaconnectors.kafka.strimzi.io created
deployment.apps/strimzi-cluster-operator created
customresourcedefinition.apiextensions.k8s.io/kafkarebalances.kafka.strimzi.io created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-global created
customresourcedefinition.apiextensions.k8s.io/kafkabridges.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkamirrormaker2s.kafka.strimzi.io created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-entity-operator-delegation created
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-client created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-client-delegation created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created
customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-broker-delegation created
configmap/strimzi-cluster-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-entity-operator created
customresourcedefinition.apiextensions.k8s.io/kafkamirrormakers.kafka.strimzi.io created
serviceaccount/strimzi-cluster-operator created

As you can tell, Strimzi created quite a few objects. Let’s wait for the operator to be ready.

$ kubectl wait deployment strimzi-cluster-operator \

-n kafka –for=condition=Available
deployment.apps/strimzi-cluster-operator condition met

Now, we can create the Kafka cluster.

$ cat <<EOF | kubectl -n kafka apply -f –
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: logscale-cluster
namespace: kafka
spec:
kafka:
version: 3.2.0
replicas: 1
listeners:
– name: plain
port: 9092
type: internal
tls: false
– name: tls
port: 9093
type: internal
tls: true
config:
offsets.topic.replication.factor: 1
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
default.replication.factor: 1
min.insync.replicas: 1
inter.broker.protocol.version: “3.2”
storage:
type: jbod
volumes:
– id: 0
type: persistent-claim
size: 10Gi
deleteClaim: false
zookeeper:
replicas: 1
storage:
type: persistent-claim
size: 1Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}
EOF

kafka.kafka.strimzi.io/logscale-cluster created

Kafka can take a few minutes to get ready, so we wait for it.

$ kubectl wait kafka/logscale-cluster \

–for=condition=Ready –timeout=300s -n kafka
kafka.kafka.strimzi.io/logscale-cluster condition met

Let’s verify Kafka is operational by sending a message (123) from a simple producer and receiving it by a simple consumer.

$ kubectl -n kafka run kafka-producer -it \

–image=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0 \

–rm=true \

–restart=Never \

— \

bin/kafka-console-producer.sh \

–bootstrap-server logscale-cluster-kafka-bootstrap:9092 \

–topic cool-topic

If you don‘t see a command prompt, try pressing enter.
>123
[2022-11-03 02:32:45,851] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 4 : {cool-topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

Don’t worry about the warnings. They are harmless. Let’s receive the 123 message:

$ kubectl -n kafka run kafka-consumer -it \

–image=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0 \

–rm=true \

–restart=Never \

— \

bin/kafka-console-consumer.sh \

–bootstrap-server logscale-cluster-kafka-bootstrap:9092 \

–topic cool-topic \

–from-beginning

If you don‘t see a command prompt, try pressing enter.
123

It works! Kafka is up and running.

Install cert-manager

Cert-manager is an X.509 certificate management solution for Kubernetes that LogScale uses as part of its security story. Let’s install it.

$ kubectl apply -f \

https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml

The manifest creates a cert-manager namespace and installs all the components there.

$ kubectl get all -n cert-manager

NAME READY STATUS RESTARTS AGE
pod/cert-manager-64d9bc8b74-g9ctl 1/1 Running 0 54s
pod/cert-manager-cainjector-6db6b64d5f-gx5jt 1/1 Running 0 54s
pod/cert-manager-webhook-6c9dd55dc8-xm2ll 1/1 Running 0 54s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cert-manager ClusterIP 10.96.160.110 <none> 9402/TCP 54s
service/cert-manager-webhook ClusterIP 10.96.127.93 <none> 443/TCP 54s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cert-manager 1/1 1 1 54s
deployment.apps/cert-manager-cainjector 1/1 1 1 54s
deployment.apps/cert-manager-webhook 1/1 1 1 54s

NAME DESIRED CURRENT READY AGE
replicaset.apps/cert-manager-64d9bc8b74 1 1 1 54s
replicaset.apps/cert-manager-cainjector-6db6b64d5f 1 1 1 54s
replicaset.apps/cert-manager-webhook-6c9dd55dc8 1 1 1 54s

Install the Humio Operator

Now that we have a functional Kafka cluster, we can install the Humio operator, which is the recommended way to install LogScale on Kubernetes. Here are some of the features of the operator:

Automates the installation of a LogScale Cluster on Kubernetes
Automates the management of LogScale Repositories, Parsers, and Ingest Tokens
Automates the management of LogScale, such as partition balancing
Automates version upgrades of LogScale
Automates configuration changes of LogScale
Allows the use various storage mediums, including hostPath or storage class PVCs
Automates cluster authentication and security, such as pod-to-pod TLS, SAML, and OAuth

First, we install the operator’s CRDs.

$ export HUMIO_OPERATOR_VERSION=0.15.0 &&\