27 April 2018

Scaling Splunk with the Qumulo File Fabric

Splunk is a market leading platform for machine data. It allows to gather all kinds of log and machine generated data in a scalable manner to index, analyze, visualize large data sets. It provides historic and real time data analytics and a large ecosystem around it, including Machine Learning libraries and many more tools.
Figure 1: Splunk harnesses machine data of any kind for indexing, searching, analysis etc.


The main components of any Splunk implementation are Forwarders, Indexers and Search Heads. Forwarders are typically software agents that run on the devices to monitor and forward steams of logs to the indexers. Indexers are the heart of Splunk’s Architecture. This is where data is parsed and
indexed in real time. Search heads are separate servers to which users connect to query data, build reports and visualize data (in smaller environments indexers and search heads can run on the same servers).

Figure 2: Splunk Architecture Components: Forwarders, Indexers, Search Heads

Data Tiering

Data in Splunk is stored in buckets:
  1. Hot Buckets: this is where data is stored at arrival. Hot buckets are kept open for writing until a certain threshold is reached. Then a hot bucket is being closed and moved to a warm bucket.
  2. Warm Buckets: Warm buckets contain are also in the index for searching and data can still be written to them. When the threshold for warm bucket capacity is reached, older warm buckets are being moved to the storage for cold buckets.
  3. Cold Buckets hold the majority of the data in most cases. Cold buckets are read only but are still in the index. Thus, cold buckets will appear in all search results, reports etc.
  4. Frozen Buckets are buckets that are not in the index anymore and are stored for archive purposes only. They are useless for searching/analysis and reporting.

Figure 3: Splunk Buckets

Qumulo Universal-Scale NAS to improve efficiency

Splunk can use local Storage or Direct Attached Storage (DAS) for all bucket types. However, this is relatively inefficient. If reliability is required, the Replication Factor (RF) and the Search Factors (SF) need to be increased. The Replication Factor indicates how many replicas are being held for the raw data while the Search Factor determines the number of copies for the index data. Both have a default value of two but can be changed at implementation time. A factor of two means that all stored data is doubled.
In addition, DAS storage is complex to manage. Whether you are using stupid JBODs or RAID arrays, in both cases there is a significant administration overhead. Rebuild times are extremely long in traditional RAID arrays which translates to increased risk of data loss.
A much better solution for the majority of data sitting in cold buckets is Qumulo’s Universal-Scale Filesystem QF2. It is a Software Defined Storage Solution that can be deployed on x64 based servers (i.e. from Qumulo and 3rd party vendors like HPE) or in the Cloud.

Qumulo’s Hypbrid Architecture

QF2 has a unique Scale-Out Architecture that starts with four nodes and it scales to many petabytes of capacity by adding nodes. It utilizes a hybrid model where SSDs are being used to build a relatively large write and read caching layer and HDDs are being used to store colder data. Thanks to this hybrid architecture, all writes and many reads are directly being served from SSDs but the economics is largely dictated by the large HDDs that Qumulo servers use.


Summary and Benefits

QF2 provides an almost bottomless pool of capacity that is extremely easy to manage
  • The capacity can be scaled as needed by adding additional nodes
  • Processing power and be scaled independently from storage. More users or more complex query will increase processing power but not storage.
  • Frozen Buckets can be avoided as data can be stored on efficient QF2 at an attractive price level in cold buckets. Data remains searchable. Storing more Splunk data allows you to run query against data covering many years of data rather than your data from the last couple of month. This provides a more accurate view of trends as well as anomalies.
  • Simplification: a Qumulo cluster is manages effortlessly compared to many DAS instances.
    Instead of increasing Splunk’s replication factor to increase availability, data in QF2 is protected by a much more efficient erasure coding.
  • Snapshots can be used to effectively backup data.


Futher detail on Qumulo’s Universal-Scale Filesystem can be found here:
The Promise of Universal Scale (White Paper). It’s a high level, marketing oriented White Paper. https://qumulo.com/documents/21/WP-Q151-Promise-of-Universal-Scale.pdf
Qumulo File Fabric Technical Overview (with good detail on data protection): https://qumulo.com/documents/20/WP-Q152-QF2-Technical-Overview.pdf


  1. Its really an Excellent post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog. Thanks for sharing....

    Carwash in omr
    usedcars in omr
    automotors in omr
    car accessories in omr
    secondhand car in omr

  2. Thank you for your post. This is excellent information. It is amazing and wonderful to visit your site.
    emc software vendors
    bmc software vendors
    Microsoft goldpartner
    sap crm service providers

  3. AWS Training in Bangalore - Live Online & Classroom
    myTectra Amazon Web Services (AWS) certification training helps you to gain real time hands on experience on AWS. myTectra offers AWS training in Bangalore using classroom and AWS Online Training globally. AWS Training at myTectra delivered by the experienced professional who has atleast 4 years of relavent AWS experince and overall 8-15 years of IT experience. myTectra Offers AWS Training since 2013 and retained the positions of Top AWS Training Company in Bangalore and India.

    IOT Training in Bangalore - Live Online & Classroom
    IOT Training course observes iot as the platform for networking of different devices on the internet and their inter related communication. Reading data through the sensors and processing it with applications sitting in the cloud and thereafter passing the processed data to generate different kind of output is the motive of the complete curricula. Students are made to understand the type of input devices and communications among the devices in a wireless media.

  4. It’s great to come across a blog every once in a while that isn’t the same out of date rehashed material. Fantastic read.

    Digital Marketing Training in Mumbai

    Six Sigma Training in Dubai

    Six Sigma Abu Dhabi

  5. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    Good discussion. Thank you.
    Six Sigma Training in Abu Dhabi
    Six Sigma Training in Dammam
    Six Sigma Training in Riyadh

  6. As you have now understood the usage of ‘Record and Playback’ tool, the following are the different posts using which you can explore the functioning of ‘Selenium IDE’
    selenium Training in chennai

  7. Thank you.Well it was nice post and very helpful information on Big Data Hadoop Online Training

  8. Al-Fares International Tents knows how significant your event is to you, either it's a Ramadan Tent or Iftar Tent, we will work with you in as large or small a capacity as you need. Ramadan Tents | Party Tents | Event Tents | Wedding Tents.
    Exhibition Tents Rental Dubai | Tent Rental Dubai | Tent Manufacturer Dubai | Tent Supplier Dubai | Tent Rental Company Dubai | Event Tent Rental Dubai | Wedding Tent Rental Dubai | Outdoor Tent Rental Sharjah | Warehouse Tent Rental Abu-Dhabi | Tents and Marquees Supplier | Tents Manufacturers UAE

    network solutions in dubai

    nas storage dubai

    SAN Solutions in Dubai