Scaling Splunk with the Qumulo File Fabric

Splunk is a market leading platform for machine data. It allows to gather all kinds of log and machine generated data in a scalable manner to index, analyze, visualize large data sets. It provides historic and real time data analytics and a large ecosystem around it, including Machine Learning libraries and many more tools.

Figure 1: Splunk harnesses machine data of any kind for indexing, searching, analysis etc.

Architecture

The main components of any Splunk implementation are Forwarders, Indexers and Search Heads. Forwarders are typically software agents that run on the devices to monitor and forward steams of logs to the indexers. Indexers are the heart of Splunk’s Architecture. This is where data is parsed and
indexed in real time. Search heads are separate servers to which users connect to query data, build reports and visualize data (in smaller environments indexers and search heads can run on the same servers).


Figure 2: Splunk Architecture Components: Forwarders, Indexers, Search Heads

Data Tiering

Data in Splunk is stored in buckets:

1. Hot Buckets: this is where data is stored at arrival. Hot buckets are kept open for writing until a certain threshold is reached. Then a hot bucket is being closed and moved to a warm bucket.
2. Warm Buckets: Warm buckets contain are also in the index for searching and data can still be written to them. When the threshold for warm bucket capacity is reached, older warm buckets are being moved to the storage for cold buckets.
3. Cold Buckets hold the majority of the data in most cases. Cold buckets are read only but are still in the index. Thus, cold buckets will appear in all search results, reports etc.
4. Frozen Buckets are buckets that are not in the index anymore and are stored for archive purposes only. They are useless for searching/analysis and reporting.
   
   

Figure 3: Splunk Buckets

Qumulo Universal-Scale NAS to improve efficiency

Splunk can use local Storage or Direct Attached Storage (DAS) for all bucket types. However, this is relatively inefficient. If reliability is required, the Replication Factor (RF) and the Search Factors (SF) need to be increased. The Replication Factor indicates how many replicas are being held for the raw data while the Search Factor determines the number of copies for the index data. Both have a default value of two but can be changed at implementation time. A factor of two means that all stored data is doubled.
In addition, DAS storage is complex to manage. Whether you are using stupid JBODs or RAID arrays, in both cases there is a significant administration overhead. Rebuild times are extremely long in traditional RAID arrays which translates to increased risk of data loss.
A much better solution for the majority of data sitting in cold buckets is Qumulo’s Universal-Scale Filesystem QF2. It is a Software Defined Storage Solution that can be deployed on x64 based servers (i.e. from Qumulo and 3rd party vendors like HPE) or in the Cloud.

Qumulo’s Hypbrid Architecture

QF2 has a unique Scale-Out Architecture that starts with four nodes and it scales to many petabytes of capacity by adding nodes. It utilizes a hybrid model where SSDs are being used to build a relatively large write and read caching layer and HDDs are being used to store colder data. Thanks to this hybrid architecture, all writes and many reads are directly being served from SSDs but the economics is largely dictated by the large HDDs that Qumulo servers use.

Summary and Benefits

QF2 provides an almost bottomless pool of capacity that is extremely easy to manage

• The capacity can be scaled as needed by adding additional nodes
• Processing power and be scaled independently from storage. More users or more complex query will increase processing power but not storage.
• Frozen Buckets can be avoided as data can be stored on efficient QF2 at an attractive price level in cold buckets. Data remains searchable. Storing more Splunk data allows you to run query against data covering many years of data rather than your data from the last couple of month. This provides a more accurate view of trends as well as anomalies.
• Simplification: a Qumulo cluster is manages effortlessly compared to many DAS instances.
  Instead of increasing Splunk’s replication factor to increase availability, data in QF2 is protected by a much more efficient erasure coding.
• Snapshots can be used to effectively backup data.

Links

Futher detail on Qumulo’s Universal-Scale Filesystem can be found here:
The Promise of Universal Scale (White Paper). It’s a high level, marketing oriented White Paper. https://qumulo.com/documents/21/WP-Q151-Promise-of-Universal-Scale.pdf
Qumulo File Fabric Technical Overview (with good detail on data protection): https://qumulo.com/documents/20/WP-Q152-QF2-Technical-Overview.pdf