EMC Isilon vs. IBM Elastic Storage Server or how IBM performs an Apple to Apple comparison

 
Readers of my blog know that I share best practices and client experiences here in my blog. In the post Isilon as a TSM Backup Target – Analyses of a Real Deployment [8], I described the “before and after” situation in an Isilon deployment for an IBM Spectrum Protect (former name Tivoli Storage Manager, TSM) solution. These results were simply a view on how a production work load looked like and how throughput and resulting backup windows evolved.

Interestingly, IBM –in the intent to position their IBM Spectrum Scale/Elastic Storage Server as the better solution-, hired a marketing firm to evaluate a performance benchmark on IBM Elastic Storage Server (ESS) against my mentioned post and publish that as a white paper [1]. The results highlighted in this paper indicate, that IBM Spectrum Scale is 11 times faster than a similar EMC Isilon configuration. I’m just guessing at why IBM did not publish that themselves, rather than paying a marketing firm to do it. I assume they are too serious to publish a comparison between snapshots of an averagely loaded production environment with a prepared benchmark test that was suited to evaluate the maximum performance of their solution.

The results published in my post were by no means showing any limits of the platform and the results were influenced by external clients, additional server and network traffic etc. Also nothing was said about server and storage utilization or any other means of potential limits or bottlenecks. It’s obvious that the authors of the white paper did not read my blog, otherwise it would be hard to explain, how they could accept such a comparison.

To wit: IBM sponsored a white paper that compared an early customer workload of a new production environment with a well prepared benchmark. They used quite different equipment but called it a “similar environment”.
 

Not a like to like workload comparison

The results published in my post are by no means showing any limits of the platform and the results were influenced by external clients, additional server and network traffic etc. Also nothing was said

How to optimize Tivoli Storage Manager operations with dsmISI and the OneFS Scale-Out File System

In a previous blog, Using Isilon as a Backup Target for TSM*), I explained why Isilon is a fantastic target for backup and archive data. While our field experiences have matured through many successful implementations, there are still things that need improvement. Especially the unawareness of TSM regarding scale-out file systems has several side effects, which I’ll explain in this blog and how they can be solved with a solution called dsmISI, developed by General Storage.

*) IBM has recently re-branded some of their products. Also the IBM Tivoli Storage Manager, TSM, got a new name. It is now called IBM Spectrum Protect (TM). However, since the name TSM is known for decades in the user community, I’ll still use it in this article.

The Manual Approach

Consider a setup illustrated in figure 1.
 
Figure 1: Two TSM Servers, connected via 10 Gigabit Ethernet to a 5 node Isilon cluster.

Backup OneFS Data over NFS/CIFS with TSM

In several of my previous posts I have mentioned the shortcomings of NDMP. One of it is the lack of support for an incremental forever strategy, a feature that TSM users typically used to have. Furthermore, TSM support for NDMP is way below average, compared to other backup Software solutions (for example, EMC Networker can create and maintain OneFS Snapshots, roll them over to tape and index them. Watch out for my blogpost here soon).

One way around this would be to backup the files in the filesystem via NFS or CIFS. To avoid the required file system scan (or treewalk), the ideal solution would be something like the isi_change_list feature mentioned in a previous post. However, the first version of that change_list API has not shown been very efficient with TSM so we have to wait for the next version which we’ll anticipate at the end of 2015. Until then, the only way to accelerate a backup via CIFS/NFS is massive parallelism. General Storage has developed a solution for this called MAGS – Massive Attack General Storage.

MAGS

During backup, TSM scans through file systems, compares their content with what it already backed up and transfers changed/new files, expires deleted files, updates properties etc. TSM is doing backups in a multi-threaded fashion, spawning off up to 128 independent threads (officially 10) doing the actual transports of data (resourceutil parameter).
However TSM does NOT multi-thread effectively when it comes to scanning a file system, hence the act of comparing existing files with backed-up versions may take a very long time –even if only little or no data changed.
Backing up an Isilon Filesystem with TSM could be as easy as entering

dsmcincremental \\my_isilon_cluster\ifs

on any TSM Windows client. Provided the appropriate permissions are in effect, this will work. But it will take a very long time. Depending on the file system structure, network latency, kind of Isilon cluster nodes and the TSM Infrastructure, there will probably be no more than 5,000 to 10,000 file system objects (files and directories) scanned per second. On an Isilon, hosting 500,000,000 file system objects, scanning alone would theoretically take about 20 hours. In real life, it usually takes much longer. Working around that “scanning” bottleneck usually involves trying to logically split the file system and backing it up with multiple jobs. So instead of running:

dsmcincremental \\my_isilon_cluster\ifs

You could run:

dsmcincremental \\my_isilon_cluster\ifs\my_first_dir
dsmcincremental \\my_isilon_cluster\ifs\my_second_dir
dsmcincremental \\my_isilon_cluster\ifs\my_third_dir
…
dsmcincremental \\my_isilon_cluster\ifs\my_nth_dir

That would certainly speed things up but you’d consider the following:

• You would have to keep track of your directories –adding a new one means it won’t get backed up unless you explicitly add it to your TSM jobs.
• It means you have to balance the number of jobs running against which directory manually. They won’t all be of the same size –there’ll be a couple very big ones and others will be small.
• It will require monitoring a potentially large number of jobs, their individual logs etc.
• It won’t take care whether your client can handle the number of parallel, memory-hungry jobs you’re starting, so you’ll constantly have to tune it yourself.

So for the time being, General Storage developed MAGS to address these issues and automated the massive parallel approach to backup. It requires one ore multiple Windows Servers where MAGS runs as a TSM client wrapper. It is started exactly like TSM’s own dsmc

magsincremental \\my_isilon_cluster\ifs

Then, MAGS performs the following steps:

• It scans the file system recursively to a configurable depth (i.e. 6 levels, which usually takes no more than a few seconds).
• It starts as many parallel TSM clients for each sub-tree found as can be handled by the machine’s memory on which it is running (maximum number of jobs and memory threshold is configurable).
• It preserves the entire file system structure in a single, consistent view. For the user running a restore via the standard TSM GUI or command line, there will only be one TSM node to address and only one tree to search.´
• It can spread its TSM Client sessions across more than one Windows machine (practical limit is about 8).
• It can be scheduled via the standard TSM client scheduler, logs to a single schedlog/errorlogand ends in a single return code.

MAGS usually shortens backup times to 10-30% of what a “normal” incremental would take, depending on the infrastructure and other bottlenecks associated with TSM servers, network etc. There are some large customers using it already and is seems to do a good job. The plan of General Storage is to include version 2.0  of Isilon’s change_list API once it is available and tested. This will the accelerate the scan-time dramatically and will most probably also reduce the  required resources on the TSM client machines.

Figure 1: Workflow of massive parallel OneFS backups with TSM using MAGS

Requirements

• At least one Windows/Intel 2008R2 or 2012 machine with at least 16 GB RAM
• Microsoft .Net4.5
• TSM Windows Backup/Archive Client V6.3 or newer
• EMC Isilon with configured SMB shares for the data to backup
• At least 3 GB of free disk space on each machine running MAGS

Impact of SSDs for meta data acceleration

File System scans and as well as all other meta data operations can be accelerated very much by using SSDs for Meta Data (Read/Write) Acceleration in  OneFS. Until recently, SSDs usage in NL-Nodes, which are typically used for backup/archive targets, has not been supported. This has not been a technical reason and EMC has recently announced that SSDs can now be used as well in NL- and HD-Nodes. This is good news because even a single SSD per Node may help to accelerate scans quite significantly. 

More Info

Thanks to Lars Henningsen from General Storage for providing the information. If you are interested in MAGS you may drop me a mail and I’ll forward to the right people or contact General Storage directly at http://www.general-storage.de

Stefan on linkedin:
https://de.linkedin.com/in/drstefanradtke/

TSM Copy pool or Isilon SyncIQ to create a copy of your backup data ?

 

If you follow this blog you are aware that many customers use Isilon Scale Out NAS as a backup target for TSM. In this context I am getting asked quite frequently whether to use Isilon’s SyncIQ or TSM Copy pools to replicate your backup data. The response to that is clear: use TSM Copy Pools. Here are the reasons:

1. SyncIQ is a parallel replication that is designed to replicate massive amount of data. It works is parallel from multiple nodes to multiple nodes at the target side. Due to the nature of a scale out NAS system it’s designed to replicate the data asynchronously (otherwise your response time on the primary side would suffer). Assume your primary site would go down due to a power outage. Transactions will not be lost due to the non-volatile NVRAM buffer that OneFS uses to store transactions. However, if you failover your cluster to the secondary side, OneFS would rollback the target directory to the last known good snapshot. That has two consequences:
   

   a.) Even if you have set the RPO to some minutes, you will lose the backup data that has been written between the last snapshot and the occurrence of the power outage (if you cannot recover your primaries site volume).
   
   b.) TSM will recognize it and would report inconsistencies between the database and the volume in the log. This might not be a disaster but you would then need to perform an AUDIT VOLUME which checks for inconsistencies between database information and a storage pool volume. This can take quite a long time !

2. The second reason why you would use TSM Copy Pools to duplicate your data is that TSM uses it intelligently in case you need to restore data in a quite utilized environment. Since TSM is aware of it’s (synchronous) Copy Pool Volume, it would mount it and use it in addition to the primary volume to restore data.
   

A final comment on disaster recovery for TSM. The above topic discusses how to make you backup data highly available but not the TSM server itself. Take a look at the General Storage Cluster Controller for TSM. It is a backup recovery solution that allows to create a highly available solution for TSM. It even covers the outage of TSM servers and works perfectly with Storage Pool Volumes on Isilon.

Resources:

[1]  White Paper: High Availability and Data Protection with EMC Isilon Scale-Out NAS
[2]  TSM Info Center: AUDIT VOLUME (Verify database information for a storage pool volume)
[3]  General Storage Cluster Controller for TSM: Brochure

Using Isilon as a Backup Target for TSM

# update 01.04.2013: based on even more existing installations we typically see more than 400 MB/s throughput with TSM on actual hardware (NL 400 nodes). That's in line with our throughput sizing for other workloads. /update

These days many customers look for efficient ways to leverage relatively new features of their backup software like de-duplication, node replication and backup of virtual environments. Several of these features cannot be efficiently used when backing up and archiving data to tape. Also we see more and more use cases that require regular and more frequent access to backup- or archived data which as well is not well suited for data that sits on tape. I am currently working with some clever guys (see section contributions) who implemented Isilon at several customers to overcome typical limits of traditional backup to tape or disk solutions and I found these ideas are worth sharing here. Although this article discusses the challenges and solutions along Tivoli Storage Manager TSM, similar principles are valid for other backup solutions of other vendors like Commvault, Symantec, EMC, CA and others. Over time, disk solutions became more and

Isilon vs. SONAS Part 6: Remote Replication

SONAS vs. Isilon Part 6: Remote Replication

In this article we’ll look at the remote replication implementations of both systems. The comparison is based on SONAS 1.4 and Isilon OneFS 7.0. Both systems provide asynchronous replication with configurable RPOs.

Use Cases

Asynchronous replication is typically used for large scale filesystems for the following use cases:

• Disaster Recovery
• Business continuance
• Disk-to-Disk backup
• Remote disk archive

Although a zero RPO cannot be achieved using asynchronous replication, it is typically feasible for non-

Isilon vs. SONAS Part 5: Multiple Authentication Sources for Multi Tenancy

Keywords: Multi tenancy, Isilon, SONAS, Authentication, Active Directory, LADAP

Article based on SONAS Version 1.3 and Isilon Version 7.0.1

For large scale NAS systems like SONAS and Isilon it would make sense for some customers, predominantly service providers, outsourcing companies and cloud providers to have a system that fully supports multi tenancy. Unfortunately the term multi tenancy is not standardizes so different people have different understanding of it. On the networking layer, we would think about separating network traffic through VLANs or encryption methods while on a server layer we think about different user roles and permissions etc. On the storage layer we care about data integrity, privacy, secure data access  and management privileges and others. And this is by far not complete but should give a little idea that multi tenancy spawns a wider area of the HW/SW stack.

Isilon vs. SONAS Part 4: Migration

# update comment April 2014: the following article misses some actual information on isi_vol_copy. isi_vol_copy is part of OneFS and allows volume based migration from Netapp and EMC NAS systems. It's very fast and handles ACL etc well. I'll do a later post on this, just cannot find the time to play with it at the moment....

One of the challenges when implementing a large scale NAS system is the migration of existing data to the

Isilon vs. SONAS Part 3: Growth Options

Isilon as well as SONAS offer node by node expansion of existing clusters. My experience from SONAS is that it takes quite a while to go through all administration steps that are required including the manual initialization of the data re-distribution. In Isilon things are little easier and adding a node is a some minutes working task.