Best Practices for Full, Differential and Incremental Backup

One of the most important decisions you will make when planning your backup strategy is whether to use full, differential, or incremental backups to protect your data.

While there are numerous variations of these, such as synthetic full, reverse incremental, and incremental forever, they are still based on these three core types.  At a high level, a full backup takes a complete copy of the data, making it easy to restore, but this option consumes a lot of storage capacity.  An incremental backup first takes a full backup, then it tracks and stores only the changes since the last backup, and then again since that last backup.

Incremental backups use much less storage capacity since they are not copying duplicate data but can take a long time to restore since the “chain” of backup files must be merged together.  A differential backup also takes a full backup first, and then it tracks the changes since the last full backup.  This uses more storage than an incremental backup as any data since the last full backup is saved. Still, it is faster to recover since it must only merge two backup files, the full backup, and the most recent differential backup.  This article will go into depth to help you understand the use cases and best practices for each.

The following table provides a summary of the tradeoffs between the three main backup types.

Characteristics	Full Backup	Incremental Backup	Differential Backup
Complexity	Easiest	Hardest	Medium
Number of Files	1	2 or more	2
Recovery Speed	Fastest	Slowest	Medium
Ease of Replication	Easiest	Hardest	Medium
Storage Capacity	Most	Smallest	Medium
Network Utilization	Most	Smallest	Medium
Backup Time	Longest	Shortest	Medium

As you can see from this table, the full backup is the easiest, yet you will need to optimize your storage capacity and network management.  The incremental backup is the slowest to recover, so you’ll need to optimize your processing power and network transfer speed during recovery.  The differential backup offers a balance between the other options.  For a thorough understanding of how backup works in an enterprise, check out Altaro’s blog on an introduction of backup and recovery.

General Backup Best Practices

For all types of backup, make sure you consider a follow these best practices:

• Make Copies of your backups in Multiple Locations – Keeping your backup files on the local server provides the benefit of a faster recovery, especially if the backup files are large. However, you should still copy your backups to shared storage (SAN) within your datacenter and also to a secondary site (or public cloud storage) through a secure connection.  This provides you with resilience if your server crashes and a disaster recovery option in the event that you have a datacenter outage.  While the data transmission to cloud storage may be slow, and you will pay for the network bandwidth and storage, it will ensure that your business can survive a major outage.  Some backup providers provide built-in tools to support copying backup files to multiple locations, which is known as duplication.
• Test Recovery on Different Hardware and Sites – Make sure you are considering the scenario where you are unable to restore a backup to the same server because the physical hardware is no longer available. Be sure that you have tested recovery to the same server, a different server, and even a server in a different datacenter to ensure that you are minimizing your downtime and maximizing your chances to recover your data successfully.
• Delete Old Backup Files – Delete files that you no longer need to free up storage capacity and reduce the security risk from data loss if your datacenter is compromised. If you operate in a regulated industry, you may be required to delete old backup after the retention period expires.  This best practice is applicable to all backup types and is often called data grooming.
• Use Solid State Drives (SSDs) – SSDs provide faster read and write speeds than traditional disks as there are no mechanical parts. If you are having trouble completing your backup during the backup window, then writing to faster media will help.  More importantly, SSDs provide a quicker recovery time as the data can be read from them much faster.
• Prioritize Backup Network Traffic / Quality of Service (QoS) – If your physical or virtual hardware supports network traffic prioritization, consider using this for your recovery traffic. You can often assign the outbound backup traffic from a server as a lower priority so that it does not interfere with production data (assuming that you can complete your backups within your backup window) and assign inbound recovery traffic as the highest priority.
• Use Compression – A majority of backup providers will include built-in compression tools so that the backup files take up less space on the disk. However, the tradeoff is that compression slows down the backup process, and decompression will slow down recovery.
• Use Encryption– Since backup files will often contain sensitive information, many organizations will want to protect them with encryption automatically. This is particularly important if those backup files are stored in a remote location or transmitted across public networks.  Be aware that this slows down the backup process, and decryption will slow down recovery.
• Use Deduplication– This feature may be offered by your backup provider, operating system, or storage vendor. It will scan your backups and detect identical files or blocks of data, and only retain one copy, removing the duplicate files.  This process can save a significant amount of storage space since many backup files contain identical and redundant content.  This will add more time to the recovery process if a deduplicated file needs to be merged with another file to recreate a full copy of the data.
• Use Item-Level Backup and Recovery – Instead of protecting and recovering an entire database, some backup ISVs will provide more granular options for specific workloads. For example, with Exchange Server, it is a common practice to offer the ability to recover a specific mailbox for a single user, rather than restoring the database with all mailboxes for all users.  This will make backup and recovery faster if only specific files need to be protected and restored.

Best Practices for Using Full Backup and Recovery

The most common type of backup is a full backup, where all the protected data is captured in its entirety in a single backup operation.  A full backup was the first type of backup created many decades ago when data was still stored on tape drives, and it simply required copying the data to other tape drives.  Since tape data had to be read sequentially, it lacked the ability to randomly access data, which is now offered with today’s hard drives.  A full backup will happen will almost always happen as a fundamental step for other backup types, including incremental and differential.  Generally, the backup software will copy all the data files, but sometimes it will include metadata, user data, or other system settings to ensure that the backup can be restored to the same (or similar) state as to when it was captured.  With the explosive growth of data in modern businesses, using only full backups probably isn’t practical due to the high costs of processing and storing it. All storage provides such as Altaro will support full backups.

Taking a full backup performs the following steps:

• The backup is requested by a user, program, or automatically on a schedule.
• The backup software will identify the type of component which needs to be protected. The disk, application, or database is usually selected by the administrator, and the backup software will determine if it must also protect any associated files.
• The backup software will wait for the component to be in a healthy state so that a complete backup can be created. This may involve pausing the component, flushing any transactions which are in memory (“quiescing”), or closing a file.  It is important that the file is in a “consistent” state so that it is healthy when it is restored.  Additionally, any other data in which the file needs will be captured, which could include metadata, system settings, boot settings, and the disk layout.
• The backup is taken.
• The backup files are stored in a designated location.
• The backup software is notified that the backup was completed successfully.

Restoring a full backup is fairly straightforward and quick and involves the following steps:

• The need for recovery is detected by a user or automatically.
• Through the backup software, the most recent (or appropriate) backup file is selected.
• The destination server or computer for the backup file is selected. The backup file will be copied here if the data is not local.
• The backup software will copy the backup file to the destination server.
• The backup will be applied, which includes restoring the data and could also include metadata, system settings, boot settings, and disk layout.
• The backup software is notified that the recovery was completed successfully.

Advantages of Using Full Backups

This section provides a list of benefits when taking a full backup.

• Reliability and Stability – Creating a full backup is straightforward, and it is stored as a single file, making it easier to manage. It is the most reliable and stable type of backup as it has the fewest components.
• Fastest Recovery Speed – Since the full backup uses only a single complete backup file, it is quickest and easiest to restore and uses the least processing power.
• Ease of Use – As the most common type of backup, it is easy to use and understand. If you are working with inexperienced users who need to manage their own backups (and recovery), then this may be the best option.  Version control is simple because it can be determined by using the timestamped on the backup file, and there is only one type of backup file.
• Recommended for Major Upgrades – If you are making a significant software or hardware change, such as updating an operating system, then always take a full backup so that you can restore to a healthy point if you need to roll back the changes.
• Ease of Replicating Backups – Since a full backup contains a single file, it is very each to copy or replicate this backup file to other locations, such as a secondary site or onto an archival tape drive.

The Disadvantage of Using Full Backups

This section provides a list of challenges to consider when taking a full backup.

• Uses More Storage Capacity – Each time you take a full backup, all of the data is captured and stored. This means that if you are taking multiple full backups at a high frequency, you could expend your storage quota quickly.  You are also likely backing up redundant copies of identical data. Make sure you are taking advantage of storage optimization techniques by using compression and deduplication.
• Uses More Network Bandwidth – Each time you create a backup, the data should be copied to a SAN or secondary site, so you must be transferred a large file through your network. Larger files mean that more data must be transmitted, so more network bandwidth is being used.
• Longer Backup Time – The duration to take a full backup is longer than other methods. This means that during this period, the computer will be allocated more computing power to the backup process, so other operations may be slower.
• Increased Need for File Security – If a full backup file is stolen, a thief can have access to the entire database. Make sure that you always encrypt your backup files.

Other Best Practices when Using Full Backups

The large size of the backup files will be your biggest challenge when using full backups, so take advantage of the storage capacity and network bandwidth optimizations mentioned earlier. Since full backups will be part of your backup strategy even if you are also using incremental or differential backups, be sure to also follow these best practices:

• Run Full Backups during Off-Hours – Backups take up a lot of processing power on your servers, and transferring the backup files over the network can consume bandwidth. Do this during off-hours to minimize the disruption of your production systems.
• Understand your Backup Windows – Make sure you know how long a full backup takes in case you need to trigger it in an emergency. For example, if you usually backup over the weekend during extended off-hours, it is important to know if it is even possible for you to take a midweek backup during a shorter backup window.

Best Practices for Using Incremental Backup and Recovery

An incremental backup will protect the data which has changed since a previous backup was taken, removing the need to take full backups each time.  This method still requires an initial full backup, but each subsequent backup is faster and uses less storage space as it only changes since the last backup is tracked and saved.  Many organizations will use the extended off-hours over the weekend to take the full backup and then take incremental backups every night.  Most leading storage provides likeAltaro will support incremental backups.

An incremental backup performs the following steps:

• The backup is requested by a user, program, or automatically on a schedule.
• The backup software will identify the type of component which needs to be protected. The disk, application, or database is usually selected by the administrator, and the backup software will know if it must also protect any associated files.
• The backup software will wait for the component to be in a healthy state so that a complete backup can be created. This may involve pausing the component, flushing any transactions which are in memory (“quiescing”), or closing a file.  It is important that the file is in a “consistent” state so that it is healthy when it is restored.  Additionally, any other data in which the file needs will be captured, which could include metadata, system settings, boot settings, and the disk layout.
• If a full backup has not been taken, then take a full backup and save the backup file using the steps above. If a full backup has been taken, then only changes since the last incremental backup will be saved in an incremental backup file.
• The full backup (if needed) and the incremental backup file is stored in a designated location.
• The backup software is notified that the backup was complete successfully.

The main downside is that when incremental backups are restored, they are much slower because each backup file must be sequentially merged together as they are structured like a “chain” of backup files.

Restoring an incremental backup involves the following steps:

• The need for recovery is detected by a user or automatically.
• Through the backup software, the most recent (or appropriate) backup file is selected. The user will usually just select the most recent timestamp, and they should not need to think about the underlying backup files.
• The destination server or computer for the backup file is selected. The backup files will be copied here if the data is not local.
• The backup software will merge all of the incremental files with the full backup file. This may happen on a dedicated backup server or on the destination server, depending on your backup software.
• The backup will be applied, which includes restoring the data and could also include metadata, system settings, boot settings, and disk layout.
• The backup software is notified that the recovery was completed successfully.

Advantages of Using Incremental Backups

This section provides a list of the main benefits of using incremental backups compared to full backups.

• Uses Less Storage Capacity – After you’ve taken the full backup, each time you take an incremental backup, only the data since the last change is captured and stored. This is the most storage efficient option of the three main backup types.
• Uses Less Network Bandwidth – After you’ve taken the full backup, each time you take an incremental backup, you must only transfer the backup file with the changed data from your servers through your network, so less network bandwidth is needed compared with the other backup types.
• Short Backup Time – After you’ve taken the full backup, each subsequent backup should be quicker as less data needs to be protected. This means that the server will be allocating less compute resources towards creating the backup.  If an application needs to be temporarily paused (quiesced), this period should also be shorter.

The disadvantage of Using Incremental Backups

This section provides a list of challenges to consider when using incremental backups compared to full backups.

• Slower Recovery Speed – The recovery of an incremental backup requires merging two or more files. The time to complete the recovery takes longer than other options. If you are merging many incremental backup files, this process could take significant time.
• Complexity of Use – Since an incremental backup requires managing multiple backup files and possibly different backup file types, it can be challenging for inexperienced users. Backup software will try to hide this complexity, but troubleshooting incremental backups can be hard.  Version control is much harder when there are multiple files with different timestamps.
• Greater Risk of Corruption – Incremental backups are restored by merging a “chain” of backup files together. However, if one of those files becomes corrupt or gets deleted, any backup files which were created after this point cannot be used.
• Complexity of Replicating Backups – Since incremental backups contain multiple files, it is more complex to copy or replicate these backup files to other locations, such as a secondary site or onto an archival tape drive. However, since each of the replicated files is smaller, the replication time should be quicker.

 

Other Best Practices when Using Incremental Backups

Recovery time will be your biggest challenge when using incremental backups, so use faster storage such as SSDs, assign extra processing power to the backup server, and prioritize network traffic.  If you decide that using incremental backups should be part of your backup strategy, then consider these best practices to follow:

• Maximize Processing Power of the Recovery Server – During recovery, incremental backups need to merge files, sometimes dozens of them. This is a compute-intensive process and takes time, which slows down the recovery.  Prioritize the backed processes for the server, which is merging these files to get a faster recovery time.
• Use a Combination of Backup Strategies – Since an incremental backup will always require a full backup, consider a hybrid strategy when you will schedule both types. Because the initial full backup may take significant time to complete, companies will often execute the full backup over a weekend, then set up incremental backups throughout the week.
• Use Different Incremental Backup Levels – Different backup providers will offer different levels of granularity. Essentially this is the smallest unit that they will backup when changed.  These can include:
  • File-Level – If any file within the backup set is changed, only that file will be incrementally backed up. This works well with small files, like office documents, but is less efficient with large files like databases.
  • Block-Level – The filesystem on a disk contains many fixed-sized blocks of data. A single file may contain multiple blocks of data.  If any of these blocks are changed, then only that part of the disk is incrementally backed up.  This is generally faster and uses less storage than backing up entire files since the blocks are smaller than files.  The main limitation is whether the backup provider supports block-level management through your filesystem and SAN.
  • Byte-Level – For an even more granular backup option, byte-level will track individual bytes of data that have changed. This results in the smallest incremental backups, which are easy to store and transmit.  This can, however, create excess processing overhead in the backup server, and not every backup provider will support it.
• Use Incremental Backup Variations – There are some more advanced backup types to consider. However, availability will vary by backup provider. The most common include:
  • Incremental Forever Backup – This option may be useful for organizations using only disks as storage media as it will not work with sequential tape drives. The backup software will only take one full backup, and then it will only take incremental backups, never taking a full backup again.  Since the data can be randomly accessed on the disk, it will also get regularly rearranged to optimize read speed.
  • Reverse Incremental Backup – If you use this variation, you will start with a full backup, then take regular incremental backups. Each incremental backup will be merged with the most recent full backup to create a new full backup.  This method lets you recover faster by always having a recent full backup available but saves on storage by not having multiple full backups.
  • Synthetic Full Backup – This variation will essentially take the full backup and any subsequent incremental backups and logically merge them, so they behave like full backups. The files are not actually merged, but the backup software is able to let users manage them like they are a single file. This is popular for organizations that have so much data that it is not realistic to take a full backup during off-hours or businesses that are always online.

Best Practices for Using Differential Backups and Recovery

Differential and incremental backups are similar since they require taking a full backup then taking smaller subsequent backups.  The main difference is that with a differential backup, each of the additional backups tracks every change since the full backup, whereas the incremental backup tracks the changes since the last incremental backup.  Most storage provides such as Altaro will support differential backups.

A differential backup performs the following steps:

• The backup is requested by a user, program, or automatically on a schedule.
• The backup software will identify the type of component which needs to be protected. The disk, application, or database is usually selected by the administrator, and the backup software will know if it must also protect any associated files.
• The backup software will wait for the component to be in a healthy state so that a complete backup can be created. This may involve pausing the component, flushing any transactions which are in memory (“quiescing”), or closing a file.  It is important that the file is in a “consistent” state so that it is healthy when it is restored.  Additionally, any other data in which the file needs will be captured, which could include metadata, system settings, boot settings, and the disk layout.
• If a full backup has not been taken, then take a full backup and save the backup file using the steps above. If a full backup has been taken, then only changes since the last full backup will be saved in a differential backup file.
• The full backup (if needed) and the differential backup file is stored in a designated location.
• The backup software is notified that the backup was complete successfully.

Differential backups can be restored quicker than incremental backups because only two backup files must be merged together.

Restoring a differential backup involves the following steps:

• The need for recovery is detected by a user or automatically.
• Through the backup software, the most recent (or appropriate) backup file is selected. The user will usually just select the most recent time; they should not need to think about the underlying backup files.
• The destination server or computer for the backup file is selected. The backup files will be copied here if the data is not local.
• The backup software will merge the selected differential file with the full backup file (1 merge between 2 files). This may happen on a dedicated backup server or on the destination server, depending on your backup software.
• The backup will be applied, which includes restoring the data and could also include metadata, system settings, boot settings, and disk layout.
• The backup software is notified that the recovery was completed successfully.

Advantages of Using Differential Backups

This section provides a list of the main benefits of using differential backups compared to full backups.

• Uses a Less Storage Capacity– After you’ve taken the full backup, each time you take a differential backup, all the data since that last full backup is captured and stored. This is more efficient than taking a full backup each time, but it will use more storage than incremental backups.
• Uses Less Network Bandwidth– After you’ve taken the full backup, each time you take a differential backup, you will transfer the changes from that last backup from your servers through your network, so less network bandwidth is needed than with using only full backups. It will use more network bandwidth than incremental backups since the files are still larger.
• Shorter Backup Time– After you’ve taken the full backup, each subsequent backup should be quicker as less data needs to be protected. This means that during this period, the computer will be allocating fewer resources creating the backup.  If an application needs to be temporarily paused (quiesced), this period should also be shorter.

The disadvantage of Using Differential Backups

This section provides a list of challenges to consider when using differential backups compared to full backups.

• Slower Recovery Speed– The recovery of a differential backup requires merging two files. The time to complete the recovery takes longer than a full backup. It is usually faster than an incremental backup, which may require merging numerous files.
• More Complex– Since a differential backup requires managing two backup files and possibly two file types, it can be challenging for inexperienced users than full backups. Backup software will try to hide this complexity, but troubleshooting differential backups can be challenging.  Version control is much harder when there are multiple files with different timestamps.
• Harder to Replicate– Since differential backups contain multiple files, it is more complex to copy or replicate these backup files to other locations, such as a secondary site or onto an archival tape drive. However, since each of the replicated files is smaller, the replication time should be quicker.

Other Best Practices when Using Differential Backups

Differential backups give you a balance between the speed of full backups and storage optimizations of incremental backups, and the best practices for each can still be applied to differential backups.  If you decide that using differential backups should be part of your backup strategy, then consider these best practices to follow:

• Use a Combination of Backup Strategies – Since a differential backup will always require a full backup, consider a hybrid strategy when you will schedule both types. Because the initial full backup may take significant time to complete, companies will often execute the full backup over a weekend, then set up differential backups throughout the week.
• Understand your Backup Windows – The file size of each differential backup will grow larger and larger with time, as more data has changed since the initial full backup was created. This means that the time the backup takes will take longer too.  If you have a strategy to run the full backup over the weekend, and differential backups each weekday, keep in mind that towards the end of the week, the backup time will take longer, so make sure that it does not exceed your backup window.

You should now have a good understanding of Full, Differential, and Incremental backups, their benefits, and their tradeoffs.  If you have any questions about these best practices or have any of your own to add, please post a comment down below!