Transparent Storage Tiering to the Cloud using Isilon Cloudpools

Isilon Storage Tiering

The Isilon Storage Tiering (aka Smartpools) is a functionality that has been around for many years. It allows to send data to specific storage pools (a storage pool is a pool of nodes with the same node type or density). This allows to store data economically effective to the lowest price level. For example (see figure 1), you may create a policy that stores all new data onto a Pool1 that has been built out of some fast S210 nodes. Response times will be extremely good, but price point is also higher (faster disks, faster CPUs etc.). Then, you create another policy that says, move all data that has not been accessed for 30 days into Pool2. This Pool2 may contain X410 nodes. Much more capacity (36 SATA disks) but somewhat lower response times compared to Pool1. Further, you may have a third pool Pool3 that contains data that has not been touched for a year. This data is hosted on HD400 nodes (very dense, 59 SATA drives + 1 SSD per 4U chassis), but portentially lower response times than tier2. However, since this tier is only used for rarely accessed data, it would not impact the user experience significantly (may vary from use case to use case of course). The movement of data, according to policies will be done by the Job-Engine in OneFS. It happens in the background and the user is not impacted. The logical location of the files (the path) will not change. That means, that we could have a single directory that contains files that reside on three different storage technologies.




Figure 1:  Policy Based Storage Tiering with Smartpools

EMC Isilon vs. IBM Elastic Storage Server or how IBM performs an Apple to Apple comparison

 
Readers of my blog know that I share best practices and client experiences here in my blog. In the post Isilon as a TSM Backup Target – Analyses of a Real Deployment [8], I described the “before and after” situation in an Isilon deployment for an IBM Spectrum Protect (former name Tivoli Storage Manager, TSM) solution. These results were simply a view on how a production work load looked like and how throughput and resulting backup windows evolved.

Interestingly, IBM –in the intent to position their IBM Spectrum Scale/Elastic Storage Server as the better solution-, hired a marketing firm to evaluate a performance benchmark on IBM Elastic Storage Server (ESS) against my mentioned post and publish that as a white paper [1]. The results highlighted in this paper indicate, that IBM Spectrum Scale is 11 times faster than a similar EMC Isilon configuration. I’m just guessing at why IBM did not publish that themselves, rather than paying a marketing firm to do it. I assume they are too serious to publish a comparison between snapshots of an averagely loaded production environment with a prepared benchmark test that was suited to evaluate the maximum performance of their solution.

The results published in my post were by no means showing any limits of the platform and the results were influenced by external clients, additional server and network traffic etc. Also nothing was said about server and storage utilization or any other means of potential limits or bottlenecks. It’s obvious that the authors of the white paper did not read my blog, otherwise it would be hard to explain, how they could accept such a comparison.

To wit: IBM sponsored a white paper that compared an early customer workload of a new production environment with a well prepared benchmark. They used quite different equipment but called it a “similar environment”.
 

Not a like to like workload comparison

The results published in my post are by no means showing any limits of the platform and the results were influenced by external clients, additional server and network traffic etc. Also nothing was said

How to optimize Tivoli Storage Manager operations with dsmISI and the OneFS Scale-Out File System

In a previous blog, Using Isilon as a Backup Target for TSM*), I explained why Isilon is a fantastic target for backup and archive data. While our field experiences have matured through many successful implementations, there are still things that need improvement. Especially the unawareness of TSM regarding scale-out file systems has several side effects, which I’ll explain in this blog and how they can be solved with a solution called dsmISI, developed by General Storage.

*) IBM has recently re-branded some of their products. Also the IBM Tivoli Storage Manager, TSM, got a new name. It is now called IBM Spectrum Protect (TM). However, since the name TSM is known for decades in the user community, I’ll still use it in this article.

The Manual Approach

Consider a setup illustrated in figure 1.
 
Figure 1: Two TSM Servers, connected via 10 Gigabit Ethernet to a 5 node Isilon cluster.

Some Aspects on Online Data vs. Tape

Technorati Tags: HDFS,Tape,TCO,Backup,Online Data,Archive

We often get involved in  discussions around cost comparison of online data vs. data that has been backed up to tape. There are tons of TCO tools that you can use which you could use to make one or the other look cheaper. I’ll give two examples here:

• An argument that is often used in favor of tape is: their acquisition cost is about 1/3 of dense archive storage. As a result, the overall TCO should be something like a third of disk.
• Well, that’s obviously just a part of the story. The real question is: how long does it take to restore business relevant data and what does it cost if you need to wait for that restore for days or weeks to complete ?

The latter aspect is no fiction. I have had customers telling me that they stopped a restore of an important database restore after eleven days. This database was just 11 TB of size but during a restore from tape you don’t have any idea how fragmented it is and what the progress on the restore is. At the end, the customer lost a lot of money because this database was not available and the company could not work on customer request. It doesn’t require a lot of creativity to think about use cases where you lose thousands of Euros every minute.

This example shows how careful you need to be when someone shows you a TCO study that proofs that one or the other technology is cheaper. You really need to understand the use case for a serious TCO calculation that includes all relevant aspects. 

However, one thing is clear on tape: the only thing that you can do with data is to restore it to disks. That’s its. And you can hope that the media will be readable after some time and that the maintenance tasks of the backup software and tape library went well to keep the data readable.

Need to analyze or access data with Hadoop ?

Furthermore, these days, companies have already realized that they have often big value in their data that they want to monetize. The keyword here is big data analytics with solutions like Hadoop. That of course is only possible if you have all your data online and accessible for relevant tools. Solution like Isilon are ideal for this purpose as they allow you to keep active data and archive data on the same platform [1]. Policies allow you to move the data to the most (cost) efficient storage media while their logical access path doesn’t change. Applications and users will find the data always on and always at the same place. Since Isilon is a multiprotocol system you can even access the data via NFS, CIFS, HDFS, Openstack Swift, FTP etc.

 

Figure 1: Multiprotocol Access to data on Isilon with policy based tiering.

I’ll probably will do another post that talks about the advantages of using Isilon for Hadoop but this one is about tape vs. online data. There are several tools out there which could be used to do your TCO calculation and I just wanted to remind you with this little article that there is more to consider than just €/TB.

 

Further Reading:

[1]  White Paper: Next Generation Storage Tiering With EMC Isilon SmartPools

[2] White Paper: Next-Generation Storage Efficiency with Isilon SmartDedupe

[3] White Paper: EMC Isilon OneFS: A Technical Overview

Direct I/O considerations for TSM Storage pools on OneFS (NFS)

 

Conext

For several reasons explained in previous posts, EMC’s Scale Out NAS system Isilon is a fantastic target for TSM storage pools. It’s easy to use; it scales linearly and provides a shared and fast filesystem for TSM instances. This makes a TSM admin’s life much easier. During a recent project we learned that the throughput we can achieve with the Tivoli Storage Manager (TSM)  vary quite significantly depending on whether TSM is configured to perform buffered vs. un-buffered I/O (direct I/O) on the storage pools that reside on OneFS. This article describes some dependencies between buffered or direct I/O, CPU utilization, I/O latencies and throughput and the role of Isilon’s Smartcache feature. Although we discuss that here in the context of TSM and OneFS, the discussed aspects should be valid for other applications as well.
 

Challenges and Options for Filesystem Backups at Petabyte Scale

The data growth for unstructured data is accelerating and filesystems that contain one or multiple petabyte are common. The current version of OneFS 7.1 has a tested support for 20PB of raw capacity, new node types and larger disks will lift that limit going forward. Challenges start even before a filesystem size reaches a petabyte. In this post I will identify the challenges that come along with such large filesystems. Even though the Isilon data protection mechanisms are very mature [1] you may want to backup your data to prevent logical or physical data due to disasters or for compliance reasons. We’ll define what the attributes of an ideal backup solution are and compare some existing technical solutions against this list of attributes and functions. The list of technical solutions is determined by my working experience and discussion with some renown colleagues from EMC’s Data Protection Availability Division (see Acknowledgement) and I would not make a claim for completeness.

 

The challenges

 

The NDMP challenge

Remember that the context of this discussion is LARGE filesystems. The industry standard solution for backing up NAS appliances is the NDMP protocol. I have mentioned several disadvantages of NDMP in a recent post. By far the biggest challenge is that NDMP does not support a progressive incremental forever strategy (well, there is an exception that I will explain later but without any 3rd party tools the statement is true). That means you need to perform a full backup every so often. In practice this is not feasible: assume we would have a dedicated 10 Gigabit Ethernet available for backup and we could saturate it with 900 MB/s. A full petabyte backup would still take

Use Isilon's new change-list feature to accelerate your backups

The new OneFS Changelist API

As mentioned in a previous post, OneFS can be completely managed through an API [1]. That's an important integration feature for environments like VMWare or interaction with other solutions.In this post is about the changelist API that it typically used by backup software to manage OneFS snaphots. Networker 8.2 and other backup software solutions for example use this API to manage and control snapshots. As we go forward we can expect that more and more data management solutions will make use of the API. However, you can also make use of the new changelist function (through API or CLI) to create a list of files that lists files which have been changed between two snapshots. If you backup software does not yet use the API itself, you can use the list and feed it into your backup client for faster backups.

This is not about NDMP

Although the new OneFS Version 7.1.1 has some new and cool NDMP features like snapshot based backup and parallel restore (see more in the OneFS 7.1.1 Release Notes [1]), many people favor to avoid NDMP at all. Here are some reasons:

Isilon as a TSM Backup Target – Analyses of a Real Deployment

In my recent blog “Using Isilon as a Backup Target” I have explained why Isilon is a perfect target for TSM backups (well, the same applies for sure to other backup and archive solutions as well but we want to show real world example here and this one has been with TSM). Beside the nice and simple administration and the fact that you can get rid of the SAN complexity for a large degree, one of the most appealing advantages is that your whole backup process becomes much faster. Why? Because

Using Isilon as a Backup Target for TSM

# update 01.04.2013: based on even more existing installations we typically see more than 400 MB/s throughput with TSM on actual hardware (NL 400 nodes). That's in line with our throughput sizing for other workloads. /update

These days many customers look for efficient ways to leverage relatively new features of their backup software like de-duplication, node replication and backup of virtual environments. Several of these features cannot be efficiently used when backing up and archiving data to tape. Also we see more and more use cases that require regular and more frequent access to backup- or archived data which as well is not well suited for data that sits on tape. I am currently working with some clever guys (see section contributions) who implemented Isilon at several customers to overcome typical limits of traditional backup to tape or disk solutions and I found these ideas are worth sharing here. Although this article discusses the challenges and solutions along Tivoli Storage Manager TSM, similar principles are valid for other backup solutions of other vendors like Commvault, Symantec, EMC, CA and others. Over time, disk solutions became more and