Isilon Backup with Networker and DD Boost to Data Domain

Quite frequently I am getting asked when we will integrate the DD-boost protocol into Isilon/OneFS for customers who want to backup to an EMC Data Domain system with high degree of deduplication. Although we haven’t got a direct boost integration into OneFS yet, there is a standard way of backing up a OneFS filesystem through DD-boost using a NetWorker Data Service Agent.

In the configuration shown below, we are leveraging NDMP to backup out of the front end of the Isilon storage platform over Ethernet to a NetWorker Storage Node. Depending on the amount of data and number of streams, you can have one or more Isilon nodes sending data and employ one or more NetWorker storage nodes to help facilitate the backup.

NetWorker features something called the Data Service Agent – commonly referred to as DSA – which is able to save or recover data via an NDMP connection between a NetWorker server or storage node and a NAS storage system. DSA essentially puts a “wrapper” around the backup data stream to enable the NetWorker software to write it to a non-NDMP backup device.

 

 

With this capability, NetWorker can then take advantage of Data Domain Boost software integrated at the NetWorker storage node. Parts of the deduplication process can then take place at the NetWorker storage node to increase throughput performance and improve network bandwidth utilization between the storage node and the Data Domain system.

The other advantage of this configuration which leverages the integration of NetWorker with DD Boost is that NetWorker can schedule and have full indexing of Data Domain replication to a second Disaster Recovery target.

In general, if each component shown here – Isilon, the Storage Node, and Data Domain – all have 10 Gigabit Ethernet links, performance of backup and recovery is optimal.  This configuration balances good throughput with the ability to leverage DD Boost integration with NetWorker. An Isilon Backup Accelerator is not required in this case because you don’t need a SAN here.

Side Node: OneFS 8.0 and Networker 9 SP1 do support NDMP Multi Stream. That yields into much faster backups as Networker can parallelize backups in a granular manner. This topic is definitely worth another blog post….

Technorati Tags: Isilon,OneFS,Backup,Networker,Data Domain,dd-boost,NDMP

EMC Isilon vs. IBM Elastic Storage Server or how IBM performs an Apple to Apple comparison

 
Readers of my blog know that I share best practices and client experiences here in my blog. In the post Isilon as a TSM Backup Target – Analyses of a Real Deployment [8], I described the “before and after” situation in an Isilon deployment for an IBM Spectrum Protect (former name Tivoli Storage Manager, TSM) solution. These results were simply a view on how a production work load looked like and how throughput and resulting backup windows evolved.

Interestingly, IBM –in the intent to position their IBM Spectrum Scale/Elastic Storage Server as the better solution-, hired a marketing firm to evaluate a performance benchmark on IBM Elastic Storage Server (ESS) against my mentioned post and publish that as a white paper [1]. The results highlighted in this paper indicate, that IBM Spectrum Scale is 11 times faster than a similar EMC Isilon configuration. I’m just guessing at why IBM did not publish that themselves, rather than paying a marketing firm to do it. I assume they are too serious to publish a comparison between snapshots of an averagely loaded production environment with a prepared benchmark test that was suited to evaluate the maximum performance of their solution.

The results published in my post were by no means showing any limits of the platform and the results were influenced by external clients, additional server and network traffic etc. Also nothing was said about server and storage utilization or any other means of potential limits or bottlenecks. It’s obvious that the authors of the white paper did not read my blog, otherwise it would be hard to explain, how they could accept such a comparison.

To wit: IBM sponsored a white paper that compared an early customer workload of a new production environment with a well prepared benchmark. They used quite different equipment but called it a “similar environment”.
 

Not a like to like workload comparison

The results published in my post are by no means showing any limits of the platform and the results were influenced by external clients, additional server and network traffic etc. Also nothing was said

Comparing Hadoop performance on DAS and Isilon and why disk locality is irrelevant

In a previous blog [3] I discussed how Isilon enables you to create a Data Lake that can serve multiple Hadoop compute clusters with different Hadoop versions and distributions simultaneously. I stated that many workloads run faster on Isilon than on traditional Hadoop clusters that use DAS storage. This statement has recently been confirmed by IDC [2] who ran various Hadoop benchmarks against Isilon and a native Hadoop cluster on DAS. Although I will show their results right at the beginning, the main purpose is to discuss why Isilon is capable of delivering such good results and what the differences are in regard to data distribution and balancing within the clusters.

The test environment

• Cloudera Hadoop Distribution CDH5.
• The Hadoop DAS cluster contained 7 nodes with one master and six worker nodes with eight 10k RPM 300 GB disks.
• The Isilon Cluster was build out of four x410 nodes, each with 57 TB disks and 3,2 TB SSDs and 10 GBE connections.
• For more details see the IDC validation report [2].

Backup OneFS Data over NFS/CIFS with TSM

In several of my previous posts I have mentioned the shortcomings of NDMP. One of it is the lack of support for an incremental forever strategy, a feature that TSM users typically used to have. Furthermore, TSM support for NDMP is way below average, compared to other backup Software solutions (for example, EMC Networker can create and maintain OneFS Snapshots, roll them over to tape and index them. Watch out for my blogpost here soon).

One way around this would be to backup the files in the filesystem via NFS or CIFS. To avoid the required file system scan (or treewalk), the ideal solution would be something like the isi_change_list feature mentioned in a previous post. However, the first version of that change_list API has not shown been very efficient with TSM so we have to wait for the next version which we’ll anticipate at the end of 2015. Until then, the only way to accelerate a backup via CIFS/NFS is massive parallelism. General Storage has developed a solution for this called MAGS – Massive Attack General Storage.

MAGS

During backup, TSM scans through file systems, compares their content with what it already backed up and transfers changed/new files, expires deleted files, updates properties etc. TSM is doing backups in a multi-threaded fashion, spawning off up to 128 independent threads (officially 10) doing the actual transports of data (resourceutil parameter).
However TSM does NOT multi-thread effectively when it comes to scanning a file system, hence the act of comparing existing files with backed-up versions may take a very long time –even if only little or no data changed.
Backing up an Isilon Filesystem with TSM could be as easy as entering

dsmcincremental \\my_isilon_cluster\ifs

on any TSM Windows client. Provided the appropriate permissions are in effect, this will work. But it will take a very long time. Depending on the file system structure, network latency, kind of Isilon cluster nodes and the TSM Infrastructure, there will probably be no more than 5,000 to 10,000 file system objects (files and directories) scanned per second. On an Isilon, hosting 500,000,000 file system objects, scanning alone would theoretically take about 20 hours. In real life, it usually takes much longer. Working around that “scanning” bottleneck usually involves trying to logically split the file system and backing it up with multiple jobs. So instead of running:

dsmcincremental \\my_isilon_cluster\ifs

You could run:

dsmcincremental \\my_isilon_cluster\ifs\my_first_dir
dsmcincremental \\my_isilon_cluster\ifs\my_second_dir
dsmcincremental \\my_isilon_cluster\ifs\my_third_dir
…
dsmcincremental \\my_isilon_cluster\ifs\my_nth_dir

That would certainly speed things up but you’d consider the following:

• You would have to keep track of your directories –adding a new one means it won’t get backed up unless you explicitly add it to your TSM jobs.
• It means you have to balance the number of jobs running against which directory manually. They won’t all be of the same size –there’ll be a couple very big ones and others will be small.
• It will require monitoring a potentially large number of jobs, their individual logs etc.
• It won’t take care whether your client can handle the number of parallel, memory-hungry jobs you’re starting, so you’ll constantly have to tune it yourself.

So for the time being, General Storage developed MAGS to address these issues and automated the massive parallel approach to backup. It requires one ore multiple Windows Servers where MAGS runs as a TSM client wrapper. It is started exactly like TSM’s own dsmc

magsincremental \\my_isilon_cluster\ifs

Then, MAGS performs the following steps:

• It scans the file system recursively to a configurable depth (i.e. 6 levels, which usually takes no more than a few seconds).
• It starts as many parallel TSM clients for each sub-tree found as can be handled by the machine’s memory on which it is running (maximum number of jobs and memory threshold is configurable).
• It preserves the entire file system structure in a single, consistent view. For the user running a restore via the standard TSM GUI or command line, there will only be one TSM node to address and only one tree to search.´
• It can spread its TSM Client sessions across more than one Windows machine (practical limit is about 8).
• It can be scheduled via the standard TSM client scheduler, logs to a single schedlog/errorlogand ends in a single return code.

MAGS usually shortens backup times to 10-30% of what a “normal” incremental would take, depending on the infrastructure and other bottlenecks associated with TSM servers, network etc. There are some large customers using it already and is seems to do a good job. The plan of General Storage is to include version 2.0  of Isilon’s change_list API once it is available and tested. This will the accelerate the scan-time dramatically and will most probably also reduce the  required resources on the TSM client machines.

Figure 1: Workflow of massive parallel OneFS backups with TSM using MAGS

Requirements

• At least one Windows/Intel 2008R2 or 2012 machine with at least 16 GB RAM
• Microsoft .Net4.5
• TSM Windows Backup/Archive Client V6.3 or newer
• EMC Isilon with configured SMB shares for the data to backup
• At least 3 GB of free disk space on each machine running MAGS

Impact of SSDs for meta data acceleration

File System scans and as well as all other meta data operations can be accelerated very much by using SSDs for Meta Data (Read/Write) Acceleration in  OneFS. Until recently, SSDs usage in NL-Nodes, which are typically used for backup/archive targets, has not been supported. This has not been a technical reason and EMC has recently announced that SSDs can now be used as well in NL- and HD-Nodes. This is good news because even a single SSD per Node may help to accelerate scans quite significantly. 

More Info

Thanks to Lars Henningsen from General Storage for providing the information. If you are interested in MAGS you may drop me a mail and I’ll forward to the right people or contact General Storage directly at http://www.general-storage.de

Stefan on linkedin:
https://de.linkedin.com/in/drstefanradtke/

TSM Copy pool or Isilon SyncIQ to create a copy of your backup data ?

 

If you follow this blog you are aware that many customers use Isilon Scale Out NAS as a backup target for TSM. In this context I am getting asked quite frequently whether to use Isilon’s SyncIQ or TSM Copy pools to replicate your backup data. The response to that is clear: use TSM Copy Pools. Here are the reasons:

1. SyncIQ is a parallel replication that is designed to replicate massive amount of data. It works is parallel from multiple nodes to multiple nodes at the target side. Due to the nature of a scale out NAS system it’s designed to replicate the data asynchronously (otherwise your response time on the primary side would suffer). Assume your primary site would go down due to a power outage. Transactions will not be lost due to the non-volatile NVRAM buffer that OneFS uses to store transactions. However, if you failover your cluster to the secondary side, OneFS would rollback the target directory to the last known good snapshot. That has two consequences:
   

   a.) Even if you have set the RPO to some minutes, you will lose the backup data that has been written between the last snapshot and the occurrence of the power outage (if you cannot recover your primaries site volume).
   
   b.) TSM will recognize it and would report inconsistencies between the database and the volume in the log. This might not be a disaster but you would then need to perform an AUDIT VOLUME which checks for inconsistencies between database information and a storage pool volume. This can take quite a long time !

2. The second reason why you would use TSM Copy Pools to duplicate your data is that TSM uses it intelligently in case you need to restore data in a quite utilized environment. Since TSM is aware of it’s (synchronous) Copy Pool Volume, it would mount it and use it in addition to the primary volume to restore data.
   

A final comment on disaster recovery for TSM. The above topic discusses how to make you backup data highly available but not the TSM server itself. Take a look at the General Storage Cluster Controller for TSM. It is a backup recovery solution that allows to create a highly available solution for TSM. It even covers the outage of TSM servers and works perfectly with Storage Pool Volumes on Isilon.

Resources:

[1]  White Paper: High Availability and Data Protection with EMC Isilon Scale-Out NAS
[2]  TSM Info Center: AUDIT VOLUME (Verify database information for a storage pool volume)
[3]  General Storage Cluster Controller for TSM: Brochure

Challenges and Options for Filesystem Backups at Petabyte Scale

The data growth for unstructured data is accelerating and filesystems that contain one or multiple petabyte are common. The current version of OneFS 7.1 has a tested support for 20PB of raw capacity, new node types and larger disks will lift that limit going forward. Challenges start even before a filesystem size reaches a petabyte. In this post I will identify the challenges that come along with such large filesystems. Even though the Isilon data protection mechanisms are very mature [1] you may want to backup your data to prevent logical or physical data due to disasters or for compliance reasons. We’ll define what the attributes of an ideal backup solution are and compare some existing technical solutions against this list of attributes and functions. The list of technical solutions is determined by my working experience and discussion with some renown colleagues from EMC’s Data Protection Availability Division (see Acknowledgement) and I would not make a claim for completeness.

 

The challenges

 

The NDMP challenge

Remember that the context of this discussion is LARGE filesystems. The industry standard solution for backing up NAS appliances is the NDMP protocol. I have mentioned several disadvantages of NDMP in a recent post. By far the biggest challenge is that NDMP does not support a progressive incremental forever strategy (well, there is an exception that I will explain later but without any 3rd party tools the statement is true). That means you need to perform a full backup every so often. In practice this is not feasible: assume we would have a dedicated 10 Gigabit Ethernet available for backup and we could saturate it with 900 MB/s. A full petabyte backup would still take

Isilon vs. SONAS Part 6: Remote Replication

SONAS vs. Isilon Part 6: Remote Replication

In this article we’ll look at the remote replication implementations of both systems. The comparison is based on SONAS 1.4 and Isilon OneFS 7.0. Both systems provide asynchronous replication with configurable RPOs.

Use Cases

Asynchronous replication is typically used for large scale filesystems for the following use cases:

• Disaster Recovery
• Business continuance
• Disk-to-Disk backup
• Remote disk archive

Although a zero RPO cannot be achieved using asynchronous replication, it is typically feasible for non-

Isilon vs. SONAS Part 5: Multiple Authentication Sources for Multi Tenancy

Keywords: Multi tenancy, Isilon, SONAS, Authentication, Active Directory, LADAP

Article based on SONAS Version 1.3 and Isilon Version 7.0.1

For large scale NAS systems like SONAS and Isilon it would make sense for some customers, predominantly service providers, outsourcing companies and cloud providers to have a system that fully supports multi tenancy. Unfortunately the term multi tenancy is not standardizes so different people have different understanding of it. On the networking layer, we would think about separating network traffic through VLANs or encryption methods while on a server layer we think about different user roles and permissions etc. On the storage layer we care about data integrity, privacy, secure data access  and management privileges and others. And this is by far not complete but should give a little idea that multi tenancy spawns a wider area of the HW/SW stack.

Isilon vs. SONAS Part 4: Migration

# update comment April 2014: the following article misses some actual information on isi_vol_copy. isi_vol_copy is part of OneFS and allows volume based migration from Netapp and EMC NAS systems. It's very fast and handles ACL etc well. I'll do a later post on this, just cannot find the time to play with it at the moment....

One of the challenges when implementing a large scale NAS system is the migration of existing data to the

Isilon vs. SONAS Part 3: Growth Options

Isilon as well as SONAS offer node by node expansion of existing clusters. My experience from SONAS is that it takes quite a while to go through all administration steps that are required including the manual initialization of the data re-distribution. In Isilon things are little easier and adding a node is a some minutes working task.