Solid State Disks under the covers

SSDs are one of the hottest technologies in storage. Made with nonvolatile flash memory, they are unencumbered by seek time and rotational latencies. From a storage administrator’s perspective, they are simply a lot faster than disk drives. However, they are far from being a “bunch of memory chips” that act like a disk drive. The challenge with flash is that individual memory cells can wear out over time, particularly if they are used for low-latency transaction processing applications. To alleviate this challenge, SSD engineers design a number of safeguards, including metadata tracking for all cells and data, compressing data to use fewer cells, parity striping to protect against cell failures, wear-leveling to use cells uniformly, “garbage collecting“ to remove obsolete data, trimming to remove deleted data, and metering to indicate when the device will stop being usable. SSDs manage everything that needs to be managed internally. Users are advised not to use defrag or other utilities that reorganize data on SSDs. They won’t perform faster, but they will wear out faster.

Source of Information : Rethinking Enterprise Storage

Virtual systems and hybrid cloud storage

IT teams use virtualization technology to consolidate, relocate, and scale applications to keep pace with the organization’s business demands and to reduce their operating costs. Hypervisors, such as ESX and ESXi from VMware and Hyper-V from Microsoft, create logical system images called virtual machines (VMs) that are independent of system hardware thereby enabling IT teams to work much more efficiently and quickly.

But virtualization creates problems for storage administrators who need more time to plan and implement changes. The storage resources for ESX and ESXi hypervisors are Virtual Machine Disk Format (VMDK) files, and for Hyper-V hypervisors, they are Virtual Hard Disk  (VHD) files. While VMs are rapidly moved from one server to another, moving the associated VMDKs and VHDs from one storage system to another is a much slower process. VMs can be relocated from one server to another without relocating the VMDKs and VHDs, but the  process of load balancing for performance usually involves shifting both VMs and VMDKS/VHDs. Data growth complicates the situation by consuming storage capacity, which degrades performance for certain VMs, and forces the IT team to move VMDKs/VHDs from one storage system to another, which can set off a chain reaction of VMDK/VHD relocations along the way. Hybrid cloud storage gracefully expands the capacity of storage, including VMDKs and VHDs, eliminating the need to move them for capacity reasons. By alleviating the pressures of data growth, hybrid cloud storage creates a more stable environment for VMs.

Source of Information : Rethinking Enterprise Storage

The constant nemesis: data growth

IDC’s Digital Universe study estimates that the amount of data stored worldwide is more than doubling every two years, so it is no surprise that managing data growth is often listed as one of the top priorities by IT leaders. IT professionals have ample experience with this problem and are well aware of the difficulties managing data growth in their corporate data centers. Balancing performance and data protection requirements with power and space constraints is a constant challenge.

IT leaders cannot surrender to the problems of data growth, so they need a strategy that will diminish the impact of it on their organizations. The hybrid cloud storage approach leverages cloud storage to offload data growth pressures to the cloud. Storage, which has always had an integral role in computing, will continue to have a fundamental role in the transformation to hybrid cloud computing—for its primary functionality (storing data) as well as its impact on those responsible for managing it.

Source of Information : Rethinking Enterprise Storage

The transformation of enterprise storage with cloud storage services

Storage has been an integral part of information technology from its inception and will continue
to be throughout the cloud computing transformation that is underway. That’s because all the data we create, use, and share has to be stored somewhere if it is to havemore than fleeting value. A lot of this data is stored in corporate data centers, but a rapidlygrowing percentage is being stored in the cloud.

Enterprise storage architectures will need to adapt to this reality and integrate with cloud storage. Just as cloud services have changed the ways we consume data, they will also change how we store, manage, and protect it. It is short-sighted to think of cloud storage merely as big disk drives in the sky when there is so much compute power in the cloud to do interesting things with it. If it is possible to find information needles in data haystacks using data analytics, it is certainly possible to discover new ways to manage all that data more effectively. For example, the implementation of erasure coding in Windows Azure Storage demonstrates how advanced error-correction technology can also be used to effectively manage cloud storage capacity.

But the advancements in enterprise storage won’t all be cloud-resident. In fact, many of the most important changes will occur in on-premises storage management functions that take advantage of hybrid cloud designs. The section “Change the architecture and change the function,” later in this chapter, examines how extending traditional storage architectures with the addition of cloud storage services makes familiar storage management functions much more powerful.

Source of Information : Rethinking Enterprise Storage

Data archiving with Azure SQL Database

The data archive mechanism with Azure SQL Database generally will be an export from Azure SQL Database. Export creates a non–transactionally consistent copy of some or all database tables using the BACPAC format (schema and data). This BACPAC file can be stored in Azure blog storage and/or downloaded to an on-premises environment and imported when desired to either Azure SQL Database or SQL Server.

To obtain a transactionally consistent export (without using third-party tools), you must either ensure that no writes occur on the source database during the export or create a database copy (which is guaranteed to be transactionally consistent).

Microsoft provides a number of options for you to use to export your data to a BACPAC file. These are as follows:

 Using the Azure portal

 Using the Export Data Tier Application Wizard in Microsoft SQL Server Management Studio, version 13.0.11000 or higher (Download SQL Server Management Studio)

 Using the SQLPackage.exe command-line utility

 Using the Start-AzureSqlDatabaseExport PowerShell cmdlet, which can be combined with the Start-AzureSqlDatabaseCopy PowerShell cmdlet

Generally, for databases smaller than 200 GB, using the Azure portal is the simplest method. If the operation goes over 20 hours, it may be cancelled. If the export does not have a clustered index, it may fail if it takes longer than 6 to 12 hours.

Source of Information : Migrating SQL Server Databases to Azure

Data archiving with SQL Server in an Azure virtual machine

The data archive mechanism with SQL Server in an Azure virtual machine generally will be a full SQL Server backup that you store for the length of time required. If you are managing your own backups, you will want to store your full database backups in Azure blob storage standard storage (or archive outside Azure). If you are not already using backup to URL to store your backups in Azure blob storage, you will want to copy from your attached disks in premium storage to Azure blob storage using standard storage.

If you are using managed backup and want to archive a full database backup for longer than 30 days, you will need to copy a full backup to a new storage location or periodically perform a full database backup (to URL) on a scheduled basis and store it in standard storage for the length of time required.

If you are using file-snapshot backup, you have another option to archive your database: you periodically can copy the backup snapshots for a given backup set to long-term storage. At any point in time, you can choose to restore to a new database when needed and create a traditional (streaming) SQL Server backup for long-term storage.

Source of Information : Migrating SQL Server Databases to Azure

Database backups with Azure SQL Database

Database backups with Azure SQL Database are performed for you automatically, and you have no option to perform manual database backups. To protect your data and enable point-in-time restore services, SQL Database takes full backups every week, multiple differential backups every day, and log backups every five minutes. Backup files are stored in geo-redundant storage with read access (RA-GRS) to ensure backups’ availability for disaster recovery purposes. The first full backup is scheduled immediately after a database is created. After the first full backup, all backups are scheduled automatically and managed silently in the background. The exact timing of full and differential backups is determined by the system to balance overall load.

SQL Database provides a point-in-time restore self-service feature for all databases regardless of service tier, but with different restore points as follows:
 Basic: Any restore point within 7 days
 Standard: Any restore point within 14 days
 Premium: Any restore point within 35 days

Point-in-time restore (geo-restore) is built on top of the SQL Database automated backup system and enables you to restore an existing or deleted database to a new database as of a specified point in time. You can restore using the Azure portal, PowerShell, or the REST API. The walk-through below will demonstrate restoring a database to a point in time using the Azure portal.

When restoring an existing database, you can restore to the same logical server or to a different server. When restoring to the same server, you create a new database. Unlike SQL Server, you cannot restore over an existing database. As a result, you will have additional storage costs while both databases exist—unless you choose to delete the existing database before restoring from the backups.

Deleting a logical server also deletes all of its databases and their database backups, after which they cannot be restored regardless of the retention period discussed previously for the automated backups. To store data for longer than the period of time provided by your service tier, you will need to export data to a BACPAC file.

Source of Information : Migrating SQL Server Databases to Azure

SQL Server Managed Backup to Windows Azure

You can use SQL Server Managed Backup to Windows Azure and have SQL Server manage and automate SQL Server backups to Azure blob storage. With this option, you specify the blob storage location and the retention period and optionally customize backup at the database level. The maximum backup size is 12.8 TB. Managed backup supports point-in-time restore during the retention period. By default, managed backups are enabled at the instance level, with the same retention period for all databases (including newly added databases). You can change the retention period on an individual database, create a custom backup schedule, or disable managed backup for an individual database using Transact-SQL. You also have the option to encrypt the database backups for additional security.

Managed backup was introduced with SQL Server 2014, and the functionality increased substantially with SQL Server 2016. For the purposes of this ebook, I will focus on SQL Server 2016.

With SQL Server 2016, if you do not specify a custom schedule, the type of backups scheduled and the backup frequency is determined based on the workload of the database.
 A full database for a database is taken in the following instances:

  •  When managed backup is enabled for the first time
  •  Log growth since the last full backup is 1 GB or more
  •  One week has passed since the last full backup
  •  The log chain is broken, such as through a manual backup

 A transaction log backup is taken in the following instances:
  •  No log backup history can be found. This usually is true when SQL Server Managed Backup to Windows Azure is enabled for the first time.
  •  The transaction log space used is 5 MB or larger.
  •  The maximum time interval of two hours since the last log backup is reached.
  •  Anytime the transaction log backup is lagging behind a full database backup. The goal is to keep the log chain ahead of full backup.
To use SQL Server managed backup, you create the following prerequisites:
 Create an Azure storage account.
 Create a container within the storage account.
 Generate a Shared Access policy and apply the Shared Access Signature (SAS) token to the Azure blob container.
 Create a SQL Server credential based on the SAS token.
CREATE CREDENTIAL WITH IDENTITY = 'SHARED ACCESS SIGNATURE', SECRET = '' ;

After satisfying these prerequisites, you use Transact-SQL to interact with SQL Server Managed Backup to Windows Azure. There are 13 system stored procedures and functions for enabling, configuring, and monitoring SQL Server Managed Backup to Windows Azure. System functions are used to retrieve existing configuration settings, parameter values, and backup file information. Each of these system stored procedures and functions provides code examples for you.
Extended events are used to surface errors and warnings. Alert mechanisms are enabled through SQL Agent jobs and SQL Server Policy-Based Management.

PowerShell cmdlets are also available to configure SQL Server Managed Backup to Windows Azure. SQL Server Management Studio supports restoring backups created by SQL Server Managed Backup to Windows Azure by using the Restore Database task.

Source of Information : Migrating SQL Server Databases to Azure

Cloud storage is for blocks too, not just files

One of the misconceptions about cloud storage is that it is only useful for storing files. This assumption comes from the popularity of file...