Scaleable backup solution for Qumulo and why NDMP is a bad choice

Modern Backup Solution options for QF2

Why NDMP is not a good solution

The Network Data Management Protocol was invented decades ago by NetApp. It addressed the fact that NAS appliances typically don’t allow 3^rd party applications -such as backup agents- to run on their operating system. Thus, the idea of NDMP was to provide a general interface that backup applications can use to backup data on NAS appliances.  Many backup software vendors have implemented it so that it could be seen as an industry standard.

However, it never really became a standard in a way that works across different Software and/or Storage vendors. Every vendor as a slightly different implementation

many things have changed and here are better options today.  In addition, the idea to separate the control and data path to offload the data traffic to the SAN is obsolete. As this was an advantage a decade ago, companies are moving away from SAN because of complexity and cost. 10, 40 and even 100 Gigabit Ethernets became the standard in datacenters so running a different network technology is no longer effective.

I’d summarize the issues that are related to NDMP as follows:

●      NDMP is not storage agnostic. In general, you cannot backup data and restore to another array from another vendor or sometimes even another OS version.

●      NDMP requires admin privileges. No problem for backups of large systems but not nice for restores, especially if a user wants to restore a single file or just a set of files.

●      Many of the backup software solutions do not index the files of the NDMP sets. May be you can store a Table of Content (TOC) with the backup but if you want to restore a single file you have to load the TOC into a temporary table to work with it. This can be very time consuming.

●      NDMP doesn't really support an incremental forever strategy. That means you have to do a full backup periodically which is a no go with large filesystems at petabyte scale that contain billions of files.

●      NDMP has been developed for Tape media in mind. So many small(er) files are being collected in a large tar file that can then be stored to tape. This is not a good idea for today’s backup targets like disk or object storage.

QF2 is a modern Scale-Out NAS solutions (and one of a few that Gartner put into their leader’s quadrant for scale out object and file solutions in 2018). It is being built for billions of files and to backup data in a modern era requires much more efficient approach.

QF2 Solution to Backup Millions of files

Any backup solution can use its native backup mechanisms to backup data on an NFS export or SMB share. By doing so, POSIX permissions or SMB ACLs will be preserved and any backup media that is supported by the backup software can be used.

If you have millions of files, you may want to backup a number of shares in parallel rather than mounting only the single root share.

While the first backup will require a full tree walk and may take quite some time to complete, Qumulo provides and elegant way to avoid treewalks and long backup times for the following incremental backups.

The Snapshot Difference API

QF 2 provides an API that can create a file list of all files that have changed between two snapshots. For example, if business wants to run a backup every day, a daily snapshot should be created by Qumulo. This is fully automated and can be schedule through the GUI, CLI or API. Then, before you start the incremental backup job, you’d call the Snapshot Differences API to pull a list of files that have changed. This list is in human and machine readable JSON format. The following picture illustrates that. In this example a new directory /mchmiel/new_dir has been created as well as the file new_file within that directory. As well, we see that the parent directory /mchmiel has change because the access time changed.

This file can easily be converted into any other format like a flat text or CSV file.

The convertred file would then be used by the backup application to only process the files within that list. Treewalks are completely avoided and future incremental backups will be very fast.

For example, with Veritas NetBackup, the relevant CLI parameter to use the list of files for the backup input is -listfile:

bpbackup -f listfile | filenames [-p policy] [-s schedule] [-S master_server...] [-t policy_type] [-L progress_log [-en]] [-w [hh:mm:ss]] [-k "keyword_phrase"]

Other vendors like Atempo have already integrated the use of Qumulo’s API into their solution so that the intermediate step to create the file can be omitted.

Advantages of the proposed Method

This method has several advantages:

• It works with all major backup solutions
• It uses the native formats of the backup solution. 
• Any media that is supported by the backup solution can be used, such as Disk, Object Storage, Tape, VTL
• Restore is very granular 
• Restore is storage agnostic
• It is very fast

Vendors have already started to integrate the Qumulo API in their backup applications. With that, the intermediate step to create the file list can be omitted.

Atempo FastScan

As mentioned above, Atempo leverages the Qumulo API already for its FastScan technology. Atempo’s FastScan feature allows to rapidly collect the list of new, changed and deleted files (by leveraging the Qumulo API) on QF2 in order to initiate data movement early on.

From a high level, Atempo’s  FastScan technology for QF2 does the following:

• Trigger and manage SnapShots used to capture coherent file lists at a given point in time.
• Retrieve the list of new, changed and deleted files since the last snapshot. 
• Perform the backup or archive to any media that is supported by Atempo (Disk, Tape, Object) without Treewalks on incremental backups. 

By using QF2’s API, Atempo implements a modern incremental forever strategy that performs backups is a very fast manner. In addition, users can restore files in a very granular manner to any filesystem with requiring administrative privileges.