19 June 2014

Isilon as a TSM Backup Target – Analyses of a Real Deployment

In my recent blog “Using Isilon as a Backup Target” I have explained why Isilon is a perfect target for TSM backups (well, the same applies for sure to other backup and archive solutions as well but we want to show real world example here and this one has been with TSM). Beside the nice and simple administration and the fact that you can get rid of the SAN complexity for a large degree, one of the most appealing advantages is that your whole backup process becomes much faster. Why? Because
Isilon allows enormous throughput. With just a five node NL400 cluster (5x 144TB raw) we do typically measure a maximum single stream throughput of ~ 600 MB/s for reads and something above 400 MB/s for writes in an ideal environment. If we consider a multi-threaded IO profile we can get a throughput of more than 2500 MB/s read and 1400 MB/s write throughput. All measured using iozone with a TSM-like workload profile: 100% sequential IO at 256kB blocksize over NFS3 and a 10 Gigabit Ethernet Link using MTU Size of 1500. Even though you may not reach these maximum throughput values in a TSM environment with thousands of threads (too many threads per node cause processing overhead), we see all TSM processes benefit from the new level of performance that Isilon delivers in real deployments:
  • Management of active data pools can be eliminated as we don’t need them anymore. If Isilon is used as a primary pool everything is accessible fast and easy. 
  • Backups and Restores will accelerate. You can configure your TSM clients to run much more threads than before. 
  • Migrations will run much faster. 
  • Reclamations will run much faster.
  • Extension of your capacity gets much easier. You just add one or more nodes to Isilon and that's it. 

The real world before and after the Isilon deployment

Now let’s look how this works by exploring the results before and after a customer deployment. Figure 1 illustrates the existing setup :

4x TSM Servers instances running on Windows 2012
2x 18 TB FC Netapp Blockstorage used as disk pool
2x TS3500 Tape Libraries with 8x LTO4 drives each

Figure 1: Existing customer setup before Isilon has been deployed

Figure2 illustrates the setup after the Isilon implementation:

4x TSM Server instances running on Windows 2012
1x 3 Node Isilon Cluster with NL Nodes ~ 260 TiB usable capacity
1x TS3500 Tape Libraries with 8x LTO4 drives each

Figure 2: Setup after the Isilon deployment

The results

Figure 3 shows the real measured results of one of the four TSM servers from May 3rd 2014 until May 16th 2014.

Figure 3: TSM1 Server processes from 03.05.2014 – 16.05.2014. 

  Observations from 3th to 12th (before Isilon deployment):

  1. Before the Isilon takeoff at the 16th, backup, archive and migration jobs ran at 100-150MB/s throughput until next day, sometimes until noon. Very short peak rates on the 7th and 09th up to 400MB/s.
  2. Archive  jobs (light blue) ran between 8 and 16 hours.
  3. Approximately same run-times for backups (dark violet) but not easy to see because covered by archive graphs.

   Observations after the Isilon deployment:

  1. Isilon went into production during the 12th. Throughput dropped on the 13th due to a miss-configured Etherchannel.
  2. Etherchannel issue corrected on the 14th. Throughput  increased to ~400 MB/s. You can see very well that the archive throughput (light blue) as well as the backup and restore throughput (violet and dark blue but covered by the archive graphs) already increased and as a result finished several hours earlier.
  3. During the 14th and the 16th, the team started to modify the TSM clients to run with more threads. This was not possible before due to the tape limitations but Isilon can ‘eat’ much more in parallel. As a result you can see that the throughput increased to 800MB/s on the 16th.
  4.  Due to the gigantic throughput, backup (dark red, mainly fileserver backup) and archive (Databases) runs have been shortened dramatically. Compare for example the archive (light blue) of the 3rd with the one of the 16th (see figure 4 which has zoomed in some interesting parts from figure 3). The throughput increased from ~150MB/s to ~750MB/s and the runtime went down from ~16 hours to ~2,5 hours. Be aware that the area under the graphs would the same in both cases assuming that the amount of data has not changed.
  5.  The migration (dark violet) has been reduced dramatically since we now store much more on Isilon and don’t need to migrate to tape that much data.
  6.  Reclamation (pink, see figure 4) has been eliminated totally (well, some days after the 16th we did see some reclamation activities reoccurring also on Isilon as well. But this is much faster because of the Isilon throughput as well as the fact the volumes are much smaller with 128 GB instead of 1,5TB for a single tape). 

Figure 4: TSM1 Server processes from 03.05-05.05.2014 and 16.05.2014 (zoomed + stretched from Fig.3).


Be aware that we only discussed the result of one out of four TSM instances here. The other three showed similar results and since the 16th, the customer is running with up to 1,4 GByte/s against the three node Isilon cluster.


This real world example shows the large improvement for backup, archive, reclamation and migration processes when using Isilon as a TSM target:
  • The average backup and archive throughputs have been increased by a factor of ~5.
  • As a result, the runtimes have been reduces by the same factor (12 hours to 2.5 hours).
  • Migrations and reclamations have been massively reduced.
  • Complexity has been reduced since all TSM Servers share a single file system. Very easy to maintain and extend. If you add a new Isilon node to the cluster, the capacity is available instantaneously for all TSM Servers with no configuration changes. The task of adding an Isilon node to the cluster takes approximately a minute or so (considering the node has been installed in the rack and cabled).
  • No more SAN components between TSM Servers and Isilon (so no more volumes, LUN-masking, SAN-Zoning, device class definition changes,…).
  • The customer was able to reduce the number of tape libraries from two to one and the number of LTO drives from 16 to 8.
  • Restores will be much faster in general.


Many thanks to Lars Henningsen and General Storage who again did an awesome job for the customer. They did the TSM Analysis and provided the data and insight while the Concat team deployed the Isilon System.

Mini FAQ

During the discussion and review of this text I got two questions which I would like to post here:

Q: Where is the TSM DB hosted in this setup? I heard storing it on Isilon is not a good idea due to latency.
A: That's correct. The databases were left untouched.Putting them on an NFS store is not the best idea. Expecially if you want to enable TSM deduplication you need to consider putting the database on Flash disks since the database will typically grow dramatically in size (instead of having an entry for every file, you a en entry for every block (if I remember correctly for every 8 or 32kB block).

Q: How can a mirrored DR solution be achieved? I heard Isilon provides asynchronous mirroring?
A: Yes, asynchronous mirror a typical way of syncing data between two scale out cluster. Be aware that we sometimes have hundreds of nodes and a potential throughput of dozens of Gigabytes per second. Copying this amount of data synchronously would introduce a massive and not acceptable latency. So for a TSM DR concept you would not use Isilon's SynIQ. What you could do instead is using a second Isilon cluster and use it for a TSM copy pool. But we aware that TSM cannot migrate a copy pool (for example to tape).

Q: Could SynIQ be used for TSM node replication?
A: Absolutely yes and it offloads the data copy process form the TSM server.



If you have similar experiences (probably using different Backup Software like EMC Networker or Commvault Simpana), questions or comments, feel free to enter them below and start the discussion.

Find me on linkedin: http://de.linkedin.com/in/drstefanradtke/


  1. Here is a direct response to this blogpost: Edison: IBM Spectrum Scale vs EMC Isilon


  2. AWS Training in Bangalore - Live Online & Classroom
    myTectra Amazon Web Services (AWS) certification training helps you to gain real time hands on experience on AWS. myTectra offers AWS training in Bangalore using classroom and AWS Online Training globally. AWS Training at myTectra delivered by the experienced professional who has atleast 4 years of relavent AWS experince and overall 8-15 years of IT experience. myTectra Offers AWS Training since 2013 and retained the positions of Top AWS Training Company in Bangalore and India.

    IOT Training in Bangalore - Live Online & Classroom
    IOT Training course observes iot as the platform for networking of different devices on the internet and their inter related communication. Reading data through the sensors and processing it with applications sitting in the cloud and thereafter passing the processed data to generate different kind of output is the motive of the complete curricula. Students are made to understand the type of input devices and communications among the devices in a wireless media.

  3. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.

    Data Science Training in Chennai
    Data science training in bangalore
    Data science online training
    Data science training in pune
    Data science training in kalyan nagar
    Data science training in Bangalore
    Data science training in tambaram

  4. A very nice guide. I will definitely follow these tips. Thank you for sharing such detailed article. I am learning a lot from you.

    java training in chennai | java training in bangalore

    java online training | java training in pune

    selenium training in chennai

    selenium training in bangalore

  5. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.

    angularjs Training in bangalore

    angularjs Training in btm

    angularjs Training in electronic-city

    angularjs Training in online

    angularjs Training in marathahalli


  6. Nice blog..! I really loved reading through this article. Thanks for sharing such a amazing post with us and keep blogging...

    Hadoop training in Hyderabad
    Hadoop online training in india
    Bigdata Hadoop online training in usa

  7. I believe there are many more pleasurable opportunities ahead for individuals that looked at your site.
    python training in OMR
    python training in tambaram
    python training in annanagar

  8. Really very nice blog information for this one and more technical skills are improve,i like that kind of post.
    Devops Training in pune


  9. A really good post,Its really very informative and interesting.it answers multiple questions that I had.Thanks a lot for sharing valuable information with us.
    aws online training

  10. Wow it is really wonderful and awesome thus it is very much useful for me to understand many concepts and helped me a lot. it is really explainable very well and i got more information from your blog.

    rpa training in velachery| rpa training in tambaram |rpa training in sholinganallur | rpa training in annanagar| rpa training in kalyannagar

  11. This post is good enough to make somebody understand this amazing thing, and I’m sure everyone will appreciate this interesting things.
    machine learning course malaysia

  12. Your work is truly appreciated round the clock and the globe. It is incredibly a comprehensive and helpful blog. VMware backup

    SAN Solutions in Dubai

  14. cool stuff you have and you keep overhaul every one of usmachine learning course malaysia