Modern Backup Solution options for QF2
Why NDMP is not a good solution
The Network Data Management Protocol was
invented decades ago by NetApp. It addressed the fact that NAS appliances
typically don’t allow 3rd party applications -such as backup agents-
to run on their operating system. Thus, the idea of NDMP was to provide a
general interface that backup applications can use to backup data on NAS
appliances. Many backup software vendors
have implemented it so that it could be seen as an industry standard.
However, it never really became a standard
in a way that works across different Software and/or Storage vendors. Every
vendor as a slightly different implementation
many things have changed and here are
better options today. In addition, the
idea to separate the control and data path to offload the data traffic to the SAN
is obsolete. As this was an advantage a decade ago, companies are moving away
from SAN because of complexity and cost. 10, 40 and even 100 Gigabit Ethernets
became the standard in datacenters so running a different network technology is
no longer effective.
I’d summarize the issues that are related
to NDMP as follows:
●
NDMP is not storage agnostic.
In general, you cannot backup data and restore to another array from another
vendor or sometimes even another OS version.
●
NDMP requires admin privileges.
No problem for backups of large systems but not nice for restores, especially
if a user wants to restore
a single file or just a set of files.
●
Many of the backup software
solutions do not index the files of the NDMP sets. May be you can store a Table
of Content (TOC) with the backup but if you want to restore a single file you
have to load the TOC into a temporary table to work with it. This can be very
time consuming.
●
NDMP doesn't really support
an incremental forever strategy.
That means you have to do a full backup periodically which is a no go with
large filesystems at petabyte scale that contain billions of files.
●
NDMP has been developed for
Tape media in mind. So many small(er) files are being collected in a large tar
file that can then be stored to tape. This is not a good idea for today’s
backup targets like disk or object storage.
QF2 is a modern Scale-Out NAS solutions
(and one of a few that Gartner put into their leader’s quadrant
for scale out object and file solutions in 2018). It is being built for
billions of files and to backup data in a modern era requires much more
efficient approach.
QF2 Solution to Backup Millions of files
Any backup solution can use its native
backup mechanisms to backup data on an NFS export or SMB share. By doing so,
POSIX permissions or SMB ACLs will be preserved and any backup media that is
supported by the backup software can be used.
If you have millions of files, you may want
to backup a number of shares in parallel rather than mounting only the single
root share.
While the first backup will require a full
tree walk and may take quite some time to complete, Qumulo provides and elegant
way to avoid treewalks and long backup times for the following incremental
backups.
The Snapshot Difference API
QF 2 provides an API that can create a file
list of all files that have changed between two snapshots. For example, if
business wants to run a backup every day, a daily snapshot should be created by
Qumulo. This is fully automated and can be schedule through the GUI, CLI or
API. Then, before you start the incremental backup job, you’d call the Snapshot
Differences API to pull a list of files that have changed. This list is in
human and machine readable JSON format. The following picture illustrates that.
In this example a new directory /mchmiel/new_dir has been created as well as the file new_file within that
directory. As well, we see that the parent directory /mchmiel has change because the access time changed.
This file can easily be converted into any
other format like a flat text or CSV file.
The convertred file would then be used by
the backup application to only process the files within that list. Treewalks
are completely avoided and future incremental backups will be very fast.
For example, with Veritas NetBackup, the
relevant CLI parameter to use the list of files for the backup input is -listfile:
bpbackup -f listfile | filenames [-p policy] [-s schedule] [-S master_server...] [-t policy_type] [-L progress_log [-en]]
[-w [hh:mm:ss]] [-k "keyword_phrase"]
Other vendors like Atempo have already
integrated the use of Qumulo’s API into their solution so that the intermediate
step to create the file can be omitted.
Advantages of the proposed Method
This method has several advantages:
- It works with all major backup solutions
- It uses the native formats of the backup solution.
- Any media that is supported by the backup solution can be used, such as Disk, Object Storage, Tape, VTL
- Restore is very granular
- Restore is storage agnostic
- It is very fast
Vendors have already started to integrate the Qumulo API in their backup applications. With that, the intermediate step to create the file list can be omitted.
Atempo FastScan
As mentioned above, Atempo leverages the Qumulo API already for its FastScan technology. Atempo’s FastScan feature allows to rapidly collect the list of new, changed and deleted files (by leveraging the Qumulo API) on QF2 in order to initiate data movement early on.
From a high level, Atempo’s FastScan technology for QF2 does the following:
- Trigger and manage SnapShots used to capture coherent file lists at a given point in time.
- Retrieve the list of new, changed and deleted files since the last snapshot.
- Perform the backup or archive to any media that is supported by Atempo (Disk, Tape, Object) without Treewalks on incremental backups.
By using QF2’s API, Atempo implements a modern incremental forever strategy that performs backups is a very fast manner. In addition, users can restore files in a very granular manner to any filesystem with requiring administrative privileges.
