About two
decades ago, a number of parallel and distributed file systems were developed. The
impetus was that, when data began growing exponentially, it became clear that
scale-out storage was the paradigm to follow for large data sets. Some examples
of good scale-out file systems are WAFL (not really scale-out) , IBM Spectrum
Scale (aka GPFS), Lustre, ZFS and OneFS. All these systems have something in
common: they had their "first boot" sometime around the year
2000.They also all have their strengths
and weaknesses. Some of these systems are not really scale-out; others are
difficult to install and operate; some require special hardware or don't
support common NAS protocols; they may have scalability limits, or lack speed
of innovation.
Just the
fact that these systems were designed 20 years ago is a problem. Many important
Internet technology trends such as DevOps, big data, converged infrastructure, containers,
IoT or virtual everything were invented much later than 2000, so these file
systems are now used in situations they were never designed to handle. It is
clearly time for a new approach to file storage
Recently, I
became aware of a modern file storage system: Qumulo File Fabric (QF2).
Gartner recently named Qumulo the only new visionary vendor in the 2017 Magic Quadrant for distributed file systems and
object storage. QF2 was designed by several of the same engineers who built
Isilon roughly 15 years ago, and obviously their experiences led them to a very
modern and flexible solution.
This article
highlights some of QF2's main features that I think are worth sharing here. 1)
QF2 is hardware independent
Several
vendors say their product is independent of hardware-specific requirements.
They may have used the term "software defined." According to Wikipedia,
two qualities of a software-defined product are:
It operates independent of any
hardware-specific dependencies and is programmatically extensible.
During an average week, an ‘interaction worker[1]’ spends 19% of the time searching and gathering information[2]. Another source specifies that in 2013 content searches cost companies over $14,000 per worker and nearly 500 hours per worker[3]. Utilizing an efficient tool to assist in this process can have a considerable ROI.
On Dell EMC’s Isilon Scale-Out NAS platforms, users generate petabytes of unstructured data and billions of files. Data is created by individual users and machine generated data is exploding due to the growing number of sensors, log files, security devices etc. To be able to mine and search within the growing data lakes (imagine, the average size on Isilon clusters is approaching 1PB!), Dell EMC is working on a Search Appliance that Indexes data on OneFS in real time and allows users and admin to search metadata and content in a fraction of a second. Alongside the functionality to increase corporate efficiency through search, we are embarking on a journey to mine and analyze user-generated data and further leverage it to create additional business-value.
In it's first version, the planned features are:
Index files from multiple Isilon clusters
Search for files by name, location, size, owner, file type, and date.
Index files within containers such as zip and tar files.
Perform a targeted full content index (FCI) on search results to view a preview of the content and search for keywords and content inside.
Perform advanced search queries including symbols, wildcards, filters, and operators.
Preview and download content.
For example, administrators and end-users can execute the following use-cases on Isilon arrays:
As an End-user, find all my MS-Word files from last year, and then index the full-content of the files, and show me all the files with ‘project Andromeda’ in it
As an End-user, show me a chart of how my files breakdown by size and/or last-accessed date and/or size
As an Admin, find all PDF files owned by corp/user1 that were modified in the first three months of this year, compact them, and export them to a specified location
As an Admin, find all MPG files that are over 1GB in the /ifs/recordings subtree
As an Admin, find all Word, Excel, and PowerPoint documents that have not been accessed in a year
To get an Idea of the capabilities of the search appliance in it’s coming first release, watch the following video.
It's a true Scale-Out Solution
The product is a virtual appliance with Wizards for configuration, and it relies on Elasticsearch indexing technology and the Lucene search engine; it has a ‘google-like’ UI with visual Filtering capabilities. The technology is scaleable: Search nodes can be added ‘hot’, and it scales to billions of files and provides responses in 1-2 seconds. Once the user filters appropriately, s/he can execute actions such as export, and full-content indexing on the results.
Real time Indexing
While the initial index scan may take some time to complete, the solution will update the index in real time by plugging into Isilon’s audit notifications and the CEE framework. The solution will index meta data such as filename, file type, path, size, dates (last modified, create, last accessed), owner/uid, group/gid and access mode. Optionally we can index full content and application specific meta data.
Fig 1: Components of the solution
It uses the OneFS API to perform certain actions like deletes and other stuff. The protocol auditing (create, delete, modify,...) forwards notifications to a CEE server (typically running on a VM) so that index updates can be made in real time (watch the video so see it). A current limitation is that only file changes carried out via SMB and NFS are monitored and updated. Changes via FTP, the OneFS API (HTTP), HDFS, or on the local file system will not be reflected in the index without a re-scan at this point in time. User actions such as downloading files that show up in a search result is performed via an SMB share.
Searches
It is important to mention that searches are done against the index and regardless of the complexity of the query; the OneFS cluster will not be affected by the search. The UI is very simple to use and allows filtering, it shows detailed metadata of search matches, visualizations and allows user actions such as preview, download, export etc.
Fig 2: The search UI
Installation
The install is self-contained. The user does not need to ‘leave’ the UI at all during the whole process.
Interested in Beta testing?
If your customer is interested in participating in the Beta test, please register here. Be aware that we are interested in serious feedback and discussion with the user. The program is not required to have a nice test and play experience.
Requirements for the Beta Test
The customer needs to provide the following to be able to run the Alpha code:
VMWare ESX v 5.x or 6.x
Resources for the VM
32GB RAM
8 vCPUs
Can be reduced for smaller Isilon clusters
556GB disk space
Can be increased up to 2TB disk space
Can be thin provisioned
2TB is enough for 6+ billion files and folders
Isilon Cluster with OneFS 7.2 or higher
Chrome or Firefox web browser (IE will be supported for GA)
External Active Directory or LDAP server(s) (optional)
The Isilon Search virtual appliance has a built-in OpenLDAP server
Add additional external AD or LDAP servers to support specific users/groups for search or administration
OneFS must expose an SMB share on /ifs. The user specified when the Isilon Search is configured must have full access to this share. The share is used to download files and access them for full content indexing
Isilon Search will automatically:
Enable protocol auditing for all Access Zones (Indexing per Access Zone is planned for a future release.
Point “Event Forwarding” to the CEE server on the Isilon Search virtual appliance
For the Beta, no existing CEE audit servers may be configured. This will not be a restriction for GA
Only one Isilon Search system can point to a single Isilon cluster
Event forwarding can only be set for one destination
How many objects on your cluster ?
To determine the total objects on the Isilon Cluster, SSH into one of the nodes and run isi job start lincount. This will return a job number. Use isi job reports view <job number> to see the results once it completes. It may take a while to complete – typically about 30m for 1 billion object (like always depending on utilization, node types etc.).
More to come
Join us for this journey of creating business-value from user-generated data. The next stations are support for additional Dell EMC platforms, and for more high-value use-cases.
References
[1] Defined by McKinsey as “high-skill knowledge workers, including managers and professionals” [2] McKinsey Global Institute (MGI) report, July 2012: “The social economy: Unlocking value and productivity through social technologies”.
[3] Source: https://aci.info/2013/11/06/the-cost-of-content-clutter-infographic/
The Isilon Storage Tiering (aka Smartpools) is a functionality that has been around for many years. It allows to send data to specific storage pools (a storage pool is a pool of nodes with the same node type or density). This allows to store data economically effective to the lowest price level. For example (see figure 1), you may create a policy that stores all new data onto a Pool1 that has been built out of some fast S210 nodes. Response times will be extremely good, but price point is also higher (faster disks, faster CPUs etc.). Then, you create another policy that says, move all data that has not been accessed for 30 days into Pool2. This Pool2 may contain X410 nodes. Much more capacity (36 SATA disks) but somewhat lower response times compared to Pool1. Further, you may have a third pool Pool3 that contains data that has not been touched for a year. This data is hosted on HD400 nodes (very dense, 59 SATA drives + 1 SSD per 4U chassis), but portentially lower response times than tier2. However, since this tier is only used for rarely accessed data, it would not impact the user experience significantly (may vary from use case to use case of course). The movement of data, according to policies will be done by the Job-Engine in OneFS. It happens in the background and the user is not impacted. The logical location of the files (the path) will not change. That means, that we could have a single directory that contains files that reside on three different storage technologies.
Figure 1: Policy Based Storage Tiering with Smartpools
Readers of my blog know that I share best practices and client experiences here in my blog. In the post Isilon as a TSM Backup Target – Analyses of a Real Deployment [8], I described the “before and after” situation in an Isilon deployment for an IBM Spectrum Protect (former name Tivoli Storage Manager, TSM) solution. These results were simply a view on how a production work load looked like and how throughput and resulting backup windows evolved.
Interestingly, IBM –in the intent to position their IBM Spectrum Scale/Elastic Storage Server as the better solution-, hired a marketing firm to evaluate a performance benchmark on IBM Elastic Storage Server (ESS) against my mentioned post and publish that as a white paper [1]. The results highlighted in this paper indicate, that IBM Spectrum Scale is 11 times faster than a similar EMC Isilon configuration. I’m just guessing at why IBM did not publish that themselves, rather than paying a marketing firm to do it. I assume they are too serious to publish a comparison between snapshots of an averagely loaded production environment with a prepared benchmark test that was suited to evaluate the maximum performance of their solution.
The results published in my post were by no means showing any limits of the platform and the results were influenced by external clients, additional server and network traffic etc. Also nothing was said about server and storage utilization or any other means of potential limits or bottlenecks. It’s obvious that the authors of the white paper did not read my blog, otherwise it would be hard to explain, how they could accept such a comparison.
To wit: IBM sponsored a white paper that compared an early customer workload of a new production environment with a well prepared benchmark. They used quite different equipment but called it a “similar environment”.
Not a like to like workload comparison
The results published in my post are by no means showing any limits of the platform and the results were influenced by external clients, additional server and network traffic etc. Also nothing was said
In a previous blog, Using Isilon as a Backup Target for TSM*), I explained why Isilon is a fantastic target for backup and archive data. While our field experiences have matured through many successful implementations, there are still things that need improvement. Especially the unawareness of TSM regarding scale-out file systems has several side effects, which I’ll explain in this blog and how they can be solved with a solution called dsmISI, developed by General Storage.
*) IBM has recently re-branded some of their products. Also the IBM Tivoli Storage Manager, TSM, got a new name. It is now called IBM Spectrum Protect (TM). However, since the name TSM is known for decades in the user community, I’ll still use it in this article.
The Manual Approach
Consider a setup illustrated in figure 1.
Figure 1: Two TSM Servers, connected via 10 Gigabit Ethernet to a 5 node Isilon cluster.
In a previous blog [3] I discussed how Isilon enables you to create a Data Lake that can serve multiple Hadoop compute clusters with different Hadoop versions and distributions simultaneously. I stated that many workloads run faster on Isilon than on traditional Hadoop clusters that use DAS storage. This statement has recently been confirmed by IDC [2] who ran various Hadoop benchmarks against Isilon and a native Hadoop cluster on DAS. Although I will show their results right at the beginning, the main purpose is to discuss why Isilon is capable of delivering such good results and what the differences are in regard to data distribution and balancing within the clusters.
The test environment
Cloudera Hadoop Distribution CDH5.
The Hadoop DAS cluster contained 7 nodes with one master and six worker nodes with eight 10k RPM 300 GB disks.
The Isilon Cluster was build out of four x410 nodes, each with 57 TB disks and 3,2 TB SSDs and 10 GBE connections.
For more details see the IDC validation report [2].
Many companies start rolling out or at least think about using Hadoop for data analysis and processing. Hadoop Distributed Filesystem (HDFS) is the underlying filesystem and typically local storage within the compute nodes is used to provide the storage capacity. HDFS has been designed to satisfy the workload characteristics for analytics and has been born during the time where 1 Gigabit Ethernet was the standard networking technology in the datacenter. The idea was to bring the data to the compute nodes in order to minimize network utilization and bandwidth and reduce latency through data locality. (I’ll post another article to discuss this in more detail and show that this requirement is less important these days where we have 10 Gigabit almost everywhere in the datacenter). For the moment we’ll look at some other side effects of this strategy which reminds me somehow to a lot of data silos. Business Intelligence dudes know what I am talking about.
Figure 1: The “Bring Data to Compute” strategy results in a lot of data silos, complex and time consuming workflows.
One thing you may already know is the fact that HDFS is not compatible with POSIX protocols like
In several of my previous posts I have mentioned the shortcomings of NDMP. One of it is the lack of support for an incremental forever strategy, a feature that TSM users typically used to have. Furthermore, TSM support for NDMP is way below average, compared to other backup Software solutions (for example, EMC Networker can create and maintain OneFS Snapshots, roll them over to tape and index them. Watch out for my blogpost here soon).
One way around this would be to backup the files in the filesystem via NFS or CIFS. To avoid the required file system scan (or treewalk), the ideal solution would be something like the isi_change_list feature mentioned in a previous post. However, the first version of that change_list API has not shown been very efficient with TSM so we have to wait for the next version which we’ll anticipate at the end of 2015. Until then, the only way to accelerate a backup via CIFS/NFS is massive parallelism. General Storage has developed a solution for this called MAGS – Massive Attack General Storage.
MAGS
During backup, TSM scans through file systems, compares their content with what it already backed up and transfers changed/new files, expires deleted files, updates properties etc. TSM is doing backups in a multi-threaded fashion, spawning off up to 128 independent threads (officially 10) doing the actual transports of data (resourceutil parameter).
However TSM does NOT multi-thread effectively when it comes to scanning a file system, hence the act of comparing existing files with backed-up versions may take a very long time –even if only little or no data changed.
Backing up an Isilon Filesystem with TSM could be as easy as entering
dsmcincremental \\my_isilon_cluster\ifs
on any TSM Windows client. Provided the appropriate permissions are in effect, this will work. But it will take a very long time. Depending on the file system structure, network latency, kind of Isilon cluster nodes and the TSM Infrastructure, there will probably be no more than 5,000 to 10,000 file system objects (files and directories) scanned per second. On an Isilon, hosting 500,000,000 file system objects, scanning alone would theoretically take about 20 hours. In real life, it usually takes much longer. Working around that “scanning” bottleneck usually involves trying to logically split the file system and backing it up with multiple jobs. So instead of running:
That would certainly speed things up but you’d consider the following:
You would have to keep track of your directories –adding a new one means it won’t get backed up unless you explicitly add it to your TSM jobs.
It means you have to balance the number of jobs running against which directory manually. They won’t all be of the same size –there’ll be a couple very big ones and others will be small.
It will require monitoring a potentially large number of jobs, their individual logs etc.
It won’t take care whether your client can handle the number of parallel, memory-hungry jobs you’re starting, so you’ll constantly have to tune it yourself.
So for the time being, General Storage developed MAGS to address these issues and automated the massive parallel approach to backup. It requires one ore multiple Windows Servers where MAGS runs as a TSM client wrapper. It is started exactly like TSM’s own dsmc
magsincremental \\my_isilon_cluster\ifs
Then, MAGS performs the following steps:
It scans the file system recursively to a configurable depth (i.e. 6 levels, which usually takes no more than a few seconds).
It starts as many parallel TSM clients for each sub-tree found as can be handled by the machine’s memory on which it is running (maximum number of jobs and memory threshold is configurable).
It preserves the entire file system structure in a single, consistent view. For the user running a restore via the standard TSM GUI or command line, there will only be one TSM node to address and only one tree to search.´
It can spread its TSM Client sessions across more than one Windows machine (practical limit is about 8).
It can be scheduled via the standard TSM client scheduler, logs to a single schedlog/errorlogand ends in a single return code.
MAGS usually shortens backup times to 10-30% of what a “normal” incremental would take, depending on the infrastructure and other bottlenecks associated with TSM servers, network etc. There are some large customers using it already and is seems to do a good job. The plan of General Storage is to include version 2.0 of Isilon’s change_list API once it is available and tested. This will the accelerate the scan-time dramatically and will most probably also reduce the required resources on the TSM client machines.
Figure 1: Workflow of massive parallel OneFS backups with TSM using MAGS
Requirements
At least one Windows/Intel 2008R2 or 2012 machine with at least 16 GB RAM
Microsoft .Net4.5
TSM Windows Backup/Archive Client V6.3 or newer
EMC Isilon with configured SMB shares for the data to backup
At least 3 GB of free disk space on each machine running MAGS
Impact of SSDs for meta data acceleration
File System scans and as well as all other meta data operations can be accelerated very much by using SSDs for Meta Data (Read/Write) Acceleration in OneFS. Until recently, SSDs usage in NL-Nodes, which are typically used for backup/archive targets, has not been supported. This has not been a technical reason and EMC has recently announced that SSDs can now be used as well in NL- and HD-Nodes. This is good news because even a single SSD per Node may help to accelerate scans quite significantly.
More Info
Thanks to Lars Henningsen from General Storage for providing the information. If you are interested in MAGS you may drop me a mail and I’ll forward to the right people or contact General Storage directly at http://www.general-storage.de
For several reasons explained in previous posts, EMC’s Scale Out NAS system Isilon is a fantastic target for TSM storage pools. It’s easy to use; it scales linearly and provides a shared and fast filesystem for TSM instances. This makes a TSM admin’s life much easier. During a recent project we learned that the throughput we can achieve with the Tivoli Storage Manager (TSM) vary quite significantly depending on whether TSM is configured to perform buffered vs. un-buffered I/O (direct I/O) on the storage pools that reside on OneFS. This article describes some dependencies between buffered or direct I/O, CPU utilization, I/O latencies and throughput and the role of Isilon’s Smartcache feature. Although we discuss that here in the context of TSM and OneFS, the discussed aspects should be valid for other applications as well.
The data growth for unstructured data is accelerating and filesystems that contain one or multiple petabyte are common. The current version of OneFS 7.1 has a tested support for 20PB of raw capacity, new node types and larger disks will lift that limit going forward. Challenges start even before a filesystem size reaches a petabyte. In this post I will identify the challenges that come along with such large filesystems. Even though the Isilon data protection mechanisms are very mature [1] you may want to backup your data to prevent logical or physical data due to disasters or for compliance reasons. We’ll define what the attributes of an ideal backup solution are and compare some existing technical solutions against this list of attributes and functions. The list of technical solutions is determined by my working experience and discussion with some renown colleagues from EMC’s Data Protection Availability Division (see Acknowledgement) and I would not make a claim for completeness.
The challenges
The NDMP challenge
Remember that the context of this discussion is LARGE filesystems. The industry standard solution for backing up NAS appliances is the NDMP protocol. I have mentioned several disadvantages of NDMP in a recent post. By far the biggest challenge is that NDMP does not support a progressive incremental forever strategy (well, there is an exception that I will explain later but without any 3rd party tools the statement is true). That means you need to perform a full backup every so often. In practice this is not feasible: assume we would have a dedicated 10 Gigabit Ethernet available for backup and we could saturate it with 900 MB/s. A full petabyte backup would still take
As mentioned in a previous post, OneFS can be completely managed through an API [1]. That's an important integration feature for environments like VMWare or interaction with other solutions.In this post is about the changelist API that it typically used by backup software to manage OneFS snaphots. Networker 8.2 and other backup software solutions for example use this API to manage and control snapshots. As we go forward we can expect that more and more data management solutions will make use of the API. However, you can also make use of the new changelist function (through API or CLI) to create a list of files that lists files which have been changed between two snapshots. If you backup software does not yet use the API itself, you can use the list and feed it into your backup client for faster backups.
This is not about NDMP
Although the new OneFS Version 7.1.1 has some new and cool NDMP features like snapshot based backup and parallel restore (see more in the OneFS 7.1.1 Release Notes [1]), many people favor to avoid NDMP at all. Here are some reasons:
In my recent blog “Using Isilon as a Backup Target” I have explained why Isilon is a perfect target for TSM backups (well, the same applies for sure to other backup and archive solutions as well but we want to show real world example here and this one has been with TSM). Beside the nice and simple administration and the fact that you can get rid of the SAN complexity for a large degree, one of the most appealing advantages is that your whole backup process becomes much faster. Why? Because
# update 01.04.2013: based on even more existing installations we typically see more than 400 MB/s throughput with TSM on actual hardware (NL 400 nodes). That's in line with our throughput sizing for other workloads. /update
These days many customers look for efficient ways to
leverage relatively new features of their backup software like de-duplication,
node replication and backup of virtual environments. Several of these features
cannot be efficiently used when backing up and archiving data to tape. Also we
see more and more use cases that require regular and more frequent access to
backup- or archived data which as well is not well suited for data that sits on
tape. I am currently working with some clever guys (see section contributions)
who implemented Isilon at several customers to overcome typical limits of
traditional backup to tape or disk solutions and I found these ideas are worth
sharing here. Although this article discusses the challenges and solutions
along Tivoli Storage Manager TSM, similar principles are valid for other backup
solutions of other vendors like Commvault, Symantec, EMC, CA and others. Over
time, disk solutions became more and
One of the
most important aspects in choosing a scale-out NAS system is the list of
supported protocols. In this post I will focus on the access protocols and give
some notes on the Hadoop Distributed Filesystem (HDFS) support in Isilon OneFS.
The supported authentication protocols were covered in this
other blog post and the management protocols may be covered in a future
post.
At this
point in time, the current Isilon operating system version is OneFS 7.1 and IBM’s
SONAS version is 1.4.2.
In this article we’ll look at the remote replication
implementations of both systems. The comparison is based on SONAS 1.4 and
Isilon OneFS 7.0. Both systems provide asynchronous replication with
configurable RPOs.
Use Cases
Asynchronous replication is typically used for large scale
filesystems for the following use cases:
Disaster Recovery
Business continuance
Disk-to-Disk backup
Remote disk archive
Although a zero RPO cannot be achieved using asynchronous
replication, it is typically feasible for non-
Keywords:
Multi tenancy, Isilon, SONAS, Authentication, Active Directory, LADAP
Article
based on SONAS Version 1.3 and Isilon Version 7.0.1
For large scale NAS systems like SONAS and Isilon it would make sense
for some customers, predominantly service providers, outsourcing companies and
cloud providers to have a system that fully supports multi tenancy. Unfortunately
the term multi tenancy is not standardizes so different people have different understanding
of it. On the networking layer, we would think about separating network traffic
through VLANs or encryption methods while on a server layer we think about different
user roles and permissions etc. On the storage layer we care about data
integrity, privacy, secure data access
and management privileges and others. And this is by far not complete
but should give a little idea that multi tenancy spawns a wider area of the
HW/SW stack.
# update comment April 2014: the following article misses some actual information on isi_vol_copy. isi_vol_copy is part of OneFS and allows volume based migration from Netapp and EMC NAS systems. It's very fast and handles ACL etc well. I'll do a later post on this, just cannot find the time to play with it at the moment....
One of the challenges when implementing a large scale NAS system is the migration of existing data to the
Isilon as well as SONAS offer node by node expansion of existing clusters. My experience from SONAS is that it takes quite a while to go through all administration steps that are required including the manual initialization of the data re-distribution. In Isilon things are little easier and adding a node is a some minutes working task.
The following discussion of the data protection mechanism refers to SONAS version 1.3 and Isilon OneFS version 6.5. To be able to understand the difference in the data protection mechanisms we need a brief look how SONAS in Isilon protect data. We’ll start with GPFS.
Here is my first Blog in 2012, just some weeks after I have started my new job at EMC. Until December 2011 I have worked 16 years for IBM and in my last job there I was in the Advanced Technical Skills team for SONAS. So the first thing I started at EMC is to dive into Isilon, an exciting solution in the high end arena of networked attached storage. You can imagine that as I am learning, I always compare in my mind things with SONAS. And my idea was to share my thoughts with the community topic by topic in a blog form rather than waiting until the end and then write a larger paper. Things may have changed then so I start right now. I have in mind covering the several topics such as:
