10 November 2018

Scaleable backup solution for Qumulo and why NDMP is a bad choice

Modern Backup Solution options for QF2


Why NDMP is not a good solution


The Network Data Management Protocol was invented decades ago by NetApp. It addressed the fact that NAS appliances typically don’t allow 3rd party applications -such as backup agents- to run on their operating system. Thus, the idea of NDMP was to provide a general interface that backup applications can use to backup data on NAS appliances.  Many backup software vendors have implemented it so that it could be seen as an industry standard.

However, it never really became a standard in a way that works across different Software and/or Storage vendors. Every vendor as a slightly different implementation
many things have changed and here are better options today.  In addition, the idea to separate the control and data path to offload the data traffic to the SAN is obsolete. As this was an advantage a decade ago, companies are moving away from SAN because of complexity and cost. 10, 40 and even 100 Gigabit Ethernets became the standard in datacenters so running a different network technology is no longer effective.

I’d summarize the issues that are related to NDMP as follows:

      NDMP is not storage agnostic. In general, you cannot backup data and restore to another array from another vendor or sometimes even another OS version.
      NDMP requires admin privileges. No problem for backups of large systems but not nice for restores, especially if a user wants to restore a single file or just a set of files.
      Many of the backup software solutions do not index the files of the NDMP sets. May be you can store a Table of Content (TOC) with the backup but if you want to restore a single file you have to load the TOC into a temporary table to work with it. This can be very time consuming.
      NDMP doesn't really support an incremental forever strategy. That means you have to do a full backup periodically which is a no go with large filesystems at petabyte scale that contain billions of files.
      NDMP has been developed for Tape media in mind. So many small(er) files are being collected in a large tar file that can then be stored to tape. This is not a good idea for today’s backup targets like disk or object storage.

QF2 is a modern Scale-Out NAS solutions (and one of a few that Gartner put into their leader’s quadrant for scale out object and file solutions in 2018). It is being built for billions of files and to backup data in a modern era requires much more efficient approach.

QF2 Solution to Backup Millions of files


Any backup solution can use its native backup mechanisms to backup data on an NFS export or SMB share. By doing so, POSIX permissions or SMB ACLs will be preserved and any backup media that is supported by the backup software can be used.

If you have millions of files, you may want to backup a number of shares in parallel rather than mounting only the single root share.

While the first backup will require a full tree walk and may take quite some time to complete, Qumulo provides and elegant way to avoid treewalks and long backup times for the following incremental backups.

The Snapshot Difference API


QF 2 provides an API that can create a file list of all files that have changed between two snapshots. For example, if business wants to run a backup every day, a daily snapshot should be created by Qumulo. This is fully automated and can be schedule through the GUI, CLI or API. Then, before you start the incremental backup job, you’d call the Snapshot Differences API to pull a list of files that have changed. This list is in human and machine readable JSON format. The following picture illustrates that. In this example a new directory /mchmiel/new_dir has been created as well as the file new_file within that directory. As well, we see that the parent directory /mchmiel has change because the access time changed.



This file can easily be converted into any other format like a flat text or CSV file.

The convertred file would then be used by the backup application to only process the files within that list. Treewalks are completely avoided and future incremental backups will be very fast.

For example, with Veritas NetBackup, the relevant CLI parameter to use the list of files for the backup input is -listfile:

bpbackup -f listfile filenames [-p policy] [-s schedule] [-S master_server...] [-t policy_type] [-L progress_log [-en]] [-w [hh:mm:ss]] [-k "keyword_phrase"]

Other vendors like Atempo have already integrated the use of Qumulo’s API into their solution so that the intermediate step to create the file can be omitted.

Advantages of the proposed Method


This method has several advantages:
  • It works with all major backup solutions
  • It uses the native formats of the backup solution. 
  • Any media that is supported by the backup solution can be used, such as Disk, Object Storage, Tape, VTL
  • Restore is very granular 
  • Restore is storage agnostic
  • It is very fast

Vendors have already started to integrate the Qumulo API in their backup applications. With that, the intermediate step to create the file list can be omitted.

Atempo FastScan


As mentioned above, Atempo leverages the Qumulo API already for its FastScan technology. Atempo’s FastScan feature allows to rapidly collect the list of new, changed and deleted files (by leveraging the Qumulo API) on QF2 in order to initiate data movement early on.

From a high level, Atempo’s  FastScan technology for QF2 does the following:

  • Trigger and manage SnapShots used to capture coherent file lists at a given point in time.
  • Retrieve the list of new, changed and deleted files since the last snapshot. 
  • Perform the backup or archive to any media that is supported by Atempo (Disk, Tape, Object) without Treewalks on incremental backups. 

By using QF2’s API, Atempo implements a modern incremental forever strategy that performs backups is a very fast manner. In addition, users can restore files in a very granular manner to any filesystem with requiring administrative privileges.


46 comments:

  1. Really fun to read this post, Stefan! Hope you're doing well.

    ReplyDelete
  2. Al-Fares International Tent company’s extensive knowledge about marquees, tents, and shades and commitment to providing our customers with high-quality products. We are excited to Kick off 2019 by introducing our services in the Middle East and Africa Regions and Market. We have been able to provide Tents, Marquees and Car Parking Shades Solution to Kingdom Saudi Arabia, Kingdom Bahrain and many other Middle East and African Countries.
    Car Parking Shades UAE | Car Parking Shades Dubai | Car Parking Shades Sharjah | Car Parking Shades Abu Dhabi | Car Parking Shades Ajman | Car Parking Shades Ras Al Khaimah

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. IMPRESSED WITH SUCH A GOOD CONTENT!!
    VERY INTERESTING
    GREAT WORK
    nas storage dubai

    ReplyDelete
  5. IMPRESSED WITH SUCH A GOOD CONTENT!!
    VERY INTERESTING
    GREAT WORK
    SAN Solutions in Dubai

    ReplyDelete
  6. Thanks for sharing such a wonderful blog...
    DevOps Training in Marathahalli - Bangalore | DevOps Training Institutes | DevOps Course Fees and Content | DevOps Interview Questions - eCare Technologies located in Marathahalli - Bangalore, is one of the best DevOps Training institute with 100% Placement support. DevOps Training in Bangalore provided by
    DevOps Certified Experts and real-time Working Professionals with handful years of experience in real time DevOps Projects.

    ReplyDelete
  7. I was basically inspecting through the web filtering for certain data and ran over your blog. I am flabbergasted by the data that you have on this blog. It shows how well you welcome this subject. Bookmarked this page, will return for extra.

    PMP

    ReplyDelete
  8. I see the best substance on your blog and I unbelievably love getting them.
    big data training

    ReplyDelete
  9. Somebody Sometimes with visits your blog normally and prescribed it as far as I can tell to peruse too.
    https://360digitmg.com/course/certification-program-in-data-science

    ReplyDelete
  10. Nice work... Much obliged for sharing this stunning and educative blog entry!
    hrdf training course

    ReplyDelete
  11. Nice work... Much obliged for sharing this stunning and educative blog entry!
    devops online training
    data science with python training

    ReplyDelete
  12. Thanks for sharing such information. Centralitytech offers managed IT services in Lexington, KY. A dedicated team of engineers remotely monitor and manage the security and health of your servers and workstations.
    online backup services

    ReplyDelete
  13. Nice post. Thank you to provide us this useful information.
    edward cullen peacoat

    ReplyDelete
  14. Online Backup Software
    Online Daily Backup software helps you to create copies of files, database, and hard drive that prevents your data loss. Click here for more information about Online Cloud Backup Reseller Program.

    ReplyDelete
  15. https://thebackupsblog.blogspot.com/2011/12/how-to-install-and-use-symantecs.html?showComment=1631102679853#c7667118404006607879

    ReplyDelete
  16. Excellent article, good concepts about Scaleable backup solution. Thanks for sharing!!!!!!!!
    We are offering 1-month free trial of backup on cloud and assuring the lowest price guarantee. Please contact us: +91-9971329945
    Visit Here-
    backup on cloud
    Web Hosting
    backup storage

    ReplyDelete
  17. This site have particular software articles which emits an impression of being a significant and significant for you individual, able software installation.
    acronis-cyber-backup-crack
    4k-software-4k-downloader-crack
    abelssoft-data2pocket-crack
    minitool-power-data-recovery-crack

    ReplyDelete
  18. Thanks admin for sharing such wonderful content on this topic. Now I have got everything I need to know. Here’s another informative content for those who looking for data security and cyber security in Lexington, KY, top cybersecurity companies. I really had a great experience with the services they provided.

    ReplyDelete
  19. Great set of tips from the master himself. Excellent ideas. Thanks for Awesome tips Keep it up
    allsoftwarepro.com
    laplink-software-pcmover-crack
    abelssoft-file-organizer-crack

    ReplyDelete
  20. This comment has been removed by the author.

    ReplyDelete
  21. Get the latest and good quality clothing items Jackets,Coats and Vests Movies, TV Series, Gaming, Casual,Faux Leather and Leather materials available Harley Quinn Live Fast Die Clown Jacket

    ReplyDelete
  22. Writers are a unique breed. You know when you're reading content written by an expert, or at least a very intelligent writer. This article is virtually perfect in my opinion.
    ikea balloon lamp

    ReplyDelete
  23. Amazing blog! I really like the way you explained such information about this post with us. And blog is really helpful for us this website
    acronis-cyber-backup-crack

    ReplyDelete
  24. Wow, amazing block structure! How long
    Have you written a blog before? Working on a blog seems easy.
    The overview of your website is pretty good, not to mention what it does.
    In the content!
    vstpatch.net
    Pianoteq Pro Crack Crack
    Xpand 2 Crack
    Acronis Cyber Backup Crack
    Plugin Boutique Scaler Crack
    Soundtoys Crack

    ReplyDelete
  25. I read this article. I think You put a lot of effort to create this article. I appreciate your work. T Birds Jacket

    ReplyDelete
  26. I read this article! I hope you will continue to have such articles to share with everyone! thank you! You can Visit my website
    https://icracker.net/https://icracker.net/

    abelssoft data2pocket crack

    bitwar data recovery crack

    3delite mp3 silence cut crack

    a4scandoc crack

    ytd video downloader pro crack

    wondershare pdfelement pro crack

    a4scandoc crack

    nordvpn crack

    ReplyDelete

  27. Just admiring your work and wondering how you managed this blog so well. It’s so remarkable that I can't afford to not go through this valuable information whenever I surf the internet! android emulators
    mipony-pro-crack
    adobe-photoshop-cc-crack
    freemake-video-converter-crack

    ReplyDelete
  28. Thanks for posting this info. I just want to let you know that I just check out your site. BTS Jungkook Apoc Jacket

    ReplyDelete
  29. online backup statistics
    Online Daily Backup software helps you to create copies of files, database, and hard drive that prevents your data loss. Click here for more information about Online Cloud Backup Reseller Program.

    ReplyDelete
  30. This article is awesome. Super immersive.

    ReplyDelete
  31. You have a fantastic website here. It's becoming more difficult to discover high-quality writing like yours these days. People like you are really valuable to me!
    Take precautions!! https://windowsactivators.org/k7-total-security-activation-key/

    ReplyDelete
  32. I am very lucky to see this grateful website. I think my needed articles here. I appreciate your good work. Keep it up. I found this great website during searching my needed articles on google. Thanks for sharing us.

    Acronis Cyber Protect Home Office Crack

    AllCrackSoft Full Version Softwares Free Download

    ReplyDelete
  33. This post is really helpful and informative. Best cyber security company offers managed IT services in Lexington, KY. A dedicated team of engineers remotely monitor and manage the security and health of your servers and workstations.
    managed IT services

    ReplyDelete
  34. I am a professional web blogger so visit my website link is given below!To get more information
    Plugin Boutique Scaler v Crack/

    ReplyDelete

  35. Really fun to read this post. Thanks for sharing this useful information

    Here is sharing some data analytics Oracle Integration Cloud Service concepts may be its helpful to you.

    Oracle Integration Cloud Service Training

    ReplyDelete