7 Things to Know About Index-free Technology
This blog was originally published Sept. 28, 2020 on humio.com. Humio is a CrowdStrike Company.
With the creation of index-free logging, log management becomes more relevant than ever for data-rich use cases because it upgrades performance for real-time and historical use cases.
Index-free logging is not just one technology based on how data is processed when it is ingested; it is the combination of several technologies including search tools. By removing indexing from the ingestion process, index-free logging speeds up search results and reduces costs. To work it requires specialized optimized search features to replace indexes.
Get a better understanding on how index-free logging has the potential to improve how developers, security professionals and analysts relate to their data by reading our 7 things to know about index-free logging.
Index free logging:
- Reduces maintenance time
- Speeds up alerts and dashboards
- Speeds up real-time search
- Enables greater compression of data
- Transfers bulk of CPU load from ingesting to searching
- Supplies users with a complete set of data
- Is ideal for high ingest rates and scalable systems
1) Reduces Maintenance Time
Because of the additional processing time and costs associated with indexing, users of indexing solutions are encouraged to curate their logs. This configuration requires the time of specialists and is unnecessary for index-free logging.
2) Speeds Up Alerts and Dashboards
In index-free logging, alerts and dashboards are processed differently than historical searches. They are handled directly in the ingest pipeline, independently of storing the data in the event store. This reduces CPU load on searching, making it only use of CPU when an actual human is doing investigation.
Other solutions use a separate metrics store (such as Prometheus) to service dashboards. Humio is unique in that it provides a pipeline to service all of your daily metrics, logs, and events in an integrated fashion.
A state machine allows the system to go from ingestion to alerting with sub-second latency. This takes pressure off dashboards and alerts, making them not searches per se, but elements of the streaming data engine. This system which side-steps ingestion delays is vital for security and development use cases in which an incident can hemorrhage quickly and increase costs and damage.
Indexing solutions are often limited in the number of real time alerts they can provide. Based on the state machine technology, index-free has virtually no limit on concurrent alerts and can run thousands of them in real time.
3) Speeds Up Real-time Search
Once users are alerted, they can begin to interact with the incident as it is happening while continuing to get live information about how the situation is evolving.
By removing the indexing step, index free logging removes time barriers to searching logs. For complex sets of data with many fields there is a high cardinality problem when it comes to indexing that can lead to slowed access to real-time search.
If there are too many value fields, indexing them all results in indexes that are significantly larger than the data you’re trying to put into the system and it takes an unreasonably long amount of time to compute this comprehensive index. Thus the data may already be irrelevant by the time it’s done being ingested.
4) Enables Greater Compression of Data
Especially in high cardinality cases, indexing adds more information to logs making them take up more space than before they were processed. Index free logging does not and instead actually compresses data significantly up to 5-20x, saving on hardware storage costs.
5) Transfers Bulk of CPU Load From Ingestion to Searching
Index based logging prioritizes historical searches over real-time, places the bulk of CPU load at the ingestion point and then having quick searches later.
Index-free logging recognizes that the load of historical searches is low in log management use cases and there is an opportunity to improve real-time search by removing indexes and prioritizing quicker ingestion. Transferring load onto the state machine for real-time alerting and queries further reduces search load to when a live human is doing actual investigation.
Humio easily accommodates these needs with optimized brute force search that uses filters and tags to reduce search space, accelerating it to typically perform as quickly as index-based search. Bloom filters exclude time segments that don’t have search results, reducing processing load. Tags additionally reduce the size of search by up to 100x.
For Cloud users, index-free logging uses multitenancy to combine the search powers of multiple organizations into one so when one active user searches, they have search speeds that exceed what they would accomplish in a self-hosted solution.
6) Supplies Users With a Complete Set of Data
Index-free logging fundamentally changes how organizations relate to their data. Optimized ingestion decreases the costs of logging everything, allowing users to log everything and use all their logs to answer novel questions long after the data was collected.
It removes the step of selecting what to include. Users get a complete view of their system with no compromises. This type of observability rewards further curiosity and inspection of data.
The relationship is no longer:“Let’s go into the logs and hope we recorded it”
It becomes: “Let’s go into the logs and we’ll find exactly what we’re looking for”
7) Is Ideal for High Ingest Rates and Scalable Systems
Powered by the flexibility of the state machine powering real-time queries and alerts and optimized for ingest, index-free logging is prepared to meet future needs as technologies in IoT and cloud computing rapidly increase log volume.
Security operation centers are acutely aware of how increasing data volumes necessitates index-free logging because they aggregate all log data from several organizations and are in danger of losing real-time observability in index-based systems due to ingest latency.
Index-free logging is prepared for this growth. Humio even offers an unlimited ingest plan that is capable of practically ingesting hundreds of terabytes a day.
Learn about Humio’s future-ready cost-saving index-free log management by requesting a free demo.
