CrowdStrike Falcon LogScale on Kubernetes

In this how-to guide, we’ll use Kubernetes to deploy a CrowdStrike Falcon® LogScale cluster on our local machine. We’ll walk through the installation of all of the prerequisite tools, the Humio operator, and then create a LogScale cluster. At each stage, we will verify that our components are up and running.
Note that our local installation of this distributed system has multiple components; in production, this would require considerable resources. This means that your local machine will need a decent amount of available CPU and memory to accommodate all the requirements. For actual sizing information and requirements, please visit this page.
Also, deploying LogScale on-premises requires a license. If you’re new to using LogScale, previously known as Humio, you can sign up for a trial license here. Licenses typically require two business days to be issued.
Finally, the content below is for demonstration purposes only and shouldn’t be used in production. Alright, let’s get to it.
Install the Prerequisite Tools
In order to prepare our local environment to run LogScale, we’ll need to install the following tools:
Let’s walk through these one at a time.
Docker
We’ll create our local Kubernetes cluster using kind, which stands for “Kubernetes in Docker.” So, we’ll need to install Docker, too.
Depending on your local machine, you have different options for a Docker solution, including:
- Rancher Desktop, which bundles in the installation of kubectl and Helm. Follow the installation instructions here.
- Colima, if you are using Mac OS. Follow the installation instructions here.
Before moving on, let’s verify that Docker is installed and operational. Use the docker version command to ensure you see a Client section and a Server section.
$ docker version Client: Version: 20.10.9 API version: 1.41 Go version: go1.16.8 Git commit: c2ea9bc Built: Thu Nov 18 21:17:06 2021 OS/Arch: darwin/arm64 Context: colima Experimental: true Server: Engine: Version: 20.10.11 API version: 1.41 (minimum version 1.12) Go version: go1.16.4 Git commit: 847da184ad5048b27f5bdf9d53d070f731b43180 Built: Fri Nov 19 03:41:34 2021 OS/Arch: linux/arm64 Experimental: false containerd: Version: v1.5.8 GitCommit: 1e5ef943eb76627a6d3b6de8cd1ef6537f393a71 runc: Version: 1.0.0-rc95 GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7 docker-init: Version: 0.19.0 GitCommit: |
It is important to configure your VM with enough resources. We recommend at least 4 CPU cores and 16 GB of memory.
kind
Kind is an official local Kubernetes cluster. It was designed for testing Kubernetes itself but is very useful for local development and testing. Follow the installation instructions here.
Next, we verify that kind was installed correctly:
$ kind version kind v0.12.0 go1.17.8 darwin/arm64 |
kubectl
Kubectl is the official CLI of Kubernetes. You can interact with any Kubernetes cluster from the command-line using kubectl (as long as you have the proper certificate). Unless your Docker installation included kubectl, you can follow the installation instructions here.
Let’s verify that kubectl was installed correctly:
$ kubectl version Client Version: version.Info{Major:”1″, Minor:”23″, GitVersion:”v1.23.4″, GitCommit:”e6c093d87ea4cbb530a7b2ae91e54c0842d8308a”, GitTreeState:”clean”, BuildDate:”2022-02-16T12:38:05Z”, GoVersion:”go1.17.7″, Compiler:”gc”, Platform:”darwin/amd64″} Server Version: version.Info{Major:”1″, Minor:”23″, GitVersion:”v1.23.4″, GitCommit:”e6c093d87ea4cbb530a7b2ae91e54c0842d8308a”, GitTreeState:”clean”, BuildDate:”2022-03-06T21:39:59Z”, GoVersion:”go1.17.7″, Compiler:”gc”, Platform:”linux/arm64″} |
If you see just a Client version, that’s not a problem. It means your kubectl is not configured to talk to any Kubernetes cluster yet. The output would look like this:
$ kubectl version I0615 18:48:57.753042 14099 versioner.go:58] invalid configuration: no configuration has been provided Client Version: version.Info{Major:”1″, Minor:”23″, GitVersion:”v1.23.4″, GitCommit:”e6c093d87ea4cbb530a7b2ae91e54c0842d8308a”, GitTreeState:”clean”, BuildDate:”2022-02-16T12:38:05Z”, GoVersion:”go1.17.7″, Compiler:”gc”, Platform:”darwin/amd64″} The connection to the server localhost:8080 was refused – did you specify the right host or port? |
Helm
Helm is the mainstream Kubernetes package manager. Follow the installation instructions here. Let’s confirm we have Helm 3 running (Helm 2 is obsolete):
$ helm version version.BuildInfo{Version:”v3.8.2″, GitCommit:”6e3701edea09e5d55a8ca2aae03a68917630e91b”, GitTreeState:”clean”, GoVersion:”go1.17.5″} |
Create a kind Cluster
With all of the prerequisites in place, let’s create our kind cluster. This step takes approximately 30 seconds.
$ kind create cluster –name logscale Creating cluster “logscale” … ✓ Ensuring node image (kindest/node:v1.23.4) 🖼 ✓ Preparing nodes 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 Set kubectl context to “kind-logscale” You can now use your cluster with: kubectl cluster-info –context kind-logscale Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂 |
Let’s verify our cluster is up and running.
$ kubectl cluster-info –context kind-logscale Kubernetes control plane is running at https://127.0.0.1:65000 CoreDNS is running at https://127.0.0.1:65000/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump’. |
Install Kafka using the Strimzi operator
Apache Kafka is required by LogScale. One of the easiest ways to install Kafka is via the Strimzi operator. Let’s install it.
First, we create a kafka namespace.
$ kubectl create ns kafka namespace/kafka created |
Then, we apply the Strimzi YAML manifest:
$ kubectl create \
-f ‘https://strimzi.io/install/latest?namespace=kafka’ \ -n kafka |
As you can tell, Strimzi created quite a few objects. Let’s wait for the operator to be ready.
$ kubectl wait deployment strimzi-cluster-operator \
-n kafka –for=condition=Available |
Now, we can create the Kafka cluster.
$ cat <<EOF | kubectl -n kafka apply -f – apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: logscale-cluster namespace: kafka spec: kafka: version: 3.2.0 replicas: 1 listeners: – name: plain port: 9092 type: internal tls: false – name: tls port: 9093 type: internal tls: true config: offsets.topic.replication.factor: 1 transaction.state.log.replication.factor: 1 transaction.state.log.min.isr: 1 default.replication.factor: 1 min.insync.replicas: 1 inter.broker.protocol.version: “3.2” storage: type: jbod volumes: – id: 0 type: persistent-claim size: 10Gi deleteClaim: false zookeeper: replicas: 1 storage: type: persistent-claim size: 1Gi deleteClaim: false entityOperator: topicOperator: {} userOperator: {} EOF kafka.kafka.strimzi.io/logscale-cluster created |
Kafka can take a few minutes to get ready, so we wait for it.
$ kubectl wait kafka/logscale-cluster \
–for=condition=Ready –timeout=300s -n kafka |
Let’s verify Kafka is operational by sending a message (123) from a simple producer and receiving it by a simple consumer.
$ kubectl -n kafka run kafka-producer -it \
–image=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0 \ –rm=true \ –restart=Never \ — \ bin/kafka-console-producer.sh \ –bootstrap-server logscale-cluster-kafka-bootstrap:9092 \ –topic cool-topic |
Don’t worry about the warnings. They are harmless. Let’s receive the 123 message:
$ kubectl -n kafka run kafka-consumer -it \
–image=quay.io/strimzi/kafka:0.32.0-kafka-3.2.0 \ –rm=true \ –restart=Never \ — \ bin/kafka-console-consumer.sh \ –bootstrap-server logscale-cluster-kafka-bootstrap:9092 \ –topic cool-topic \ –from-beginning |
It works! Kafka is up and running.
Install cert-manager
Cert-manager is an X.509 certificate management solution for Kubernetes that LogScale uses as part of its security story. Let’s install it.
$ kubectl apply -f \
https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml |
The manifest creates a cert-manager namespace and installs all the components there.
$ kubectl get all -n cert-manager
NAME READY STATUS RESTARTS AGE |
Install the Humio Operator
Now that we have a functional Kafka cluster, we can install the Humio operator, which is the recommended way to install LogScale on Kubernetes. Here are some of the features of the operator:
- Automates the installation of a LogScale Cluster on Kubernetes
- Automates the management of LogScale Repositories, Parsers, and Ingest Tokens
- Automates the management of LogScale, such as partition balancing
- Automates version upgrades of LogScale
- Automates configuration changes of LogScale
- Allows the use various storage mediums, including hostPath or storage class PVCs
- Automates cluster authentication and security, such as pod-to-pod TLS, SAML, and OAuth
First, we install the operator’s CRDs.
$ export HUMIO_OPERATOR_VERSION=0.15.0 &&\
kubectl apply –server-side -f \ https://raw.githubusercontent.com/humio/humio-operator/humio-operator-${HUMIO_OPERATOR_VERSION}/config/crd/bases/core.humio.com_humioclusters.yaml &&\ kubectl apply –server-side -f \ https://raw.githubusercontent.com/humio/humio-operator/humio-operator-${HUMIO_OPERATOR_VERSION}/config/crd/bases/core.humio.com_humioexternalclusters.yaml &&\ kubectl apply –server-side -f \ https://raw.githubusercontent.com/humio/humio-operator/humio-operator-${HUMIO_OPERATOR_VERSION}/config/crd/bases/core.humio.com_humioingesttokens.yaml &&\ kubectl apply –server-side -f \ https://raw.githubusercontent.com/humio/humio-operator/humio-operator-${HUMIO_OPERATOR_VERSION}/config/crd/bases/core.humio.com_humioparsers.yaml &&\ kubectl apply –server-side -f \ https://raw.githubusercontent.com/humio/humio-operator/humio-operator-${HUMIO_OPERATOR_VERSION}/config/crd/bases/core.humio.com_humiorepositories.yaml &&\ kubectl apply –server-side -f \ https://raw.githubusercontent.com/humio/humio-operator/humio-operator-${HUMIO_OPERATOR_VERSION}/config/crd/bases/core.humio.com_humioviews.yaml &&\ kubectl apply –server-side -f \ https://raw.githubusercontent.com/humio/humio-operator/humio-operator-${HUMIO_OPERATOR_VERSION}/config/crd/bases/core.humio.com_humioalerts.yaml &&\ kubectl apply –server-side -f \ https://raw.githubusercontent.com/humio/humio-operator/humio-operator-${HUMIO_OPERATOR_VERSION}/config/crd/bases/core.humio.com_humioactions.yaml
|
Next, we add the Humio helm repository.
$ helm repo add humio-operator https://humio.github.io/humio-operator “humio-operator” has been added to your repositories |
Now, we’re ready to install the Humio operator.
$ helm install humio-operator humio-operator/humio-operator \ –namespace logging \ –create-namespace \ –version=“${HUMIO_OPERATOR_VERSION}“ NAME: humio-operator LAST DEPLOYED: Thu Nov 3 10:43:49 2022 NAMESPACE: logging STATUS: deployed REVISION: 1 TEST SUITE: None |
Finally, everything is in place, and it’s time to create the LogScale cluster.
Create the LogScale Cluster
To create the LogScale cluster, we apply a HumioCluster CRD. There are many configuration options for resource limits, requests, and replication factors. For local testing on kind, we should go for the lower end and make sure we don’t exceed the capacity of the VM supporting your kind cluster.
First, let’s create a namespace called logscale.
$ kubectl create ns logscale namespace/logscale created |
As we mentioned above, running LogScale on kind will require a license.
Once we have the license, we create a Kubernetes secret that contains the license (replace the <REDACTED> part in the end with your license data).
$ kubectl create secret generic logscale-trial-license -n logscale \
–from-literal=data=<REDACTED> |
The next thing we need to find out is the service names for ZooKeeper and Kafka, as we’ll use those when setting up our LogScale cluster.
$ kubectl get svc -n kafka NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE logscale-cluster-kafka-bootstrap ClusterIP 10.96.127.36 <none> 9091/TCP,9092/TCP,9093/TCP 5h50m logscale-cluster-kafka-brokers ClusterIP None <none> 9090/TCP,9091/TCP,9092/TCP,9093/TCP 5h50m logscale-cluster-zookeeper-client ClusterIP 10.96.170.109 <none> 2181/TCP 5h52m logscale-cluster-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 5h52m |
We use the logscale-cluster-kafka-brokers and logscale-cluster-zookeeper-client service names, along with ports, in the configuration of the LogScale cluster as the KAFKA_SERVERS and ZOOKEEPER_URL.
Now, we can create the LogScale cluster itself.
$ cat <<EOF | kubectl -n logscale apply -f – apiVersion: core.humio.com/v1alpha1 kind: HumioCluster metadata: name: logscale-cluster spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: – matchExpressions: – key: humio_node_type operator: In values: – core – matchExpressions: – key: kubernetes.io/arch operator: In values: – arm64 – matchExpressions: – key: kubernetes.io/os operator: In values: – linux license: secretKeyRef: name: logscale-trial-license key: data image: “humio/humio-core:1.56.3” nodeCount: 1 tls: enabled: false targetReplicationFactor: 1 storagePartitionsCount: 24 digestPartitionsCount: 24 resources: limits: cpu: “2” memory: 4Gi requests: cpu: “1” memory: 2Gi dataVolumePersistentVolumeClaimSpecTemplate: storageClassName: standard accessModes: [ReadWriteOnce] resources: requests: storage: 10Gi environmentVariables: – name: “HUMIO_MEMORY_OPTS” value: “-Xss2m -Xms1g -Xmx2g -XX:MaxDirectMemorySize=1g” – name: “ZOOKEEPER_URL” value: logscale-cluster-zookeeper-client.kafka.svc.cluster.local:2181 – name: “KAFKA_SERVERS” value: logscale-cluster-kafka-brokers.kafka.svc.cluster.local:9092 – name: AUTHENTICATION_METHOD value: “single-user” – name: SINGLE_USER_PASSWORD value: “password” EOF humiocluster.core.humio.com/logscale-cluster created |
Let’s check the status of our LogScale cluster.
$ kubectl get humiocluster logscale-cluster -n logscale NAME STATE NODES VERSION logscale-cluster Running 1 1.36.1–build-124825–sha-6402d163827020e288913da5ea6441e07946e57e |
We’re up and running!
Test the LogScale Cluster
We can check out the LogScale web UI by doing a port-forward.
$ kubectl -n logscale port-forward svc/logscale-cluster 8080 Forwarding from 127.0.0.1:8080 -> 8080 Forwarding from [::1]:8080 -> 8080 |
Now, we can browse to https://www.crowdstrike.com:8080, log in, and play with the LogScale dashboard.
If you’re following along with the steps, then congratulations! 🎉 You now have a working LogScale cluster running locally on a kind cluster.
Next Steps
At this point, you can follow the interactive LogScale tutorial at https://www.crowdstrike.com:8080/tutorial, which uses the built-in Sandbox repository.
With LogScale up and running, you can start instrumenting your systems to start sending log data to LogScale. LogScale has integrations with several log shippers to make it easy to aggregate logs from all of your different sources.
Conclusion
LogScale is a sophisticated platform for consuming and monitoring event data ingested from logs. It is designed for large-scale environments, composed of multiple components, and has non-trivial dependencies like Kafka, ZooKeeper and cert-manager.
In this article, we walked through an entire deployment of LogScale on a local Kubernetes cluster using the Humio operator for Kubernetes. We started from scratch and then:
- Installed all the necessary prerequisite tools
- Created a kind cluster
- Installed Kafka using the Strimzi operator
- Installed cert-manager
- Installed the Humio operator
Lastly, we created a LogScale cluster and accessed it through its web UI.
Now, you’re fully equipped to learn and play with LogScale in the comfort of your own machine. Go for it!
Content provided by Grant Schofield