Here’s a quick tip from the Head GEEK to help you Restore MS Exchange 2007 Using CDP Enterprise Edition.

R1Soft CDP Enterprise Edition for Windows uses MS Volume Shadow Copy Service (VSS) to protect your Windows server and applications like Exchange.

You can restore your MS Exchange Storage Groups in 5 Easy Steps:

1. Use CDP to Restore the Exchange data and log files.
2. Use Exchange Eseutil to replay your log files.
3. Create an Exchange Recovery Group using the Exchange Management Console.
4. Copy the Exchange data and log files you restored to the Exchange Recovery Storage Group directory.
5. Mount the Storage Group.

Tip KB article including step-by-step screenshots is here.

Have questions about our High-Performance CDP products? Just email headgeek (at) r1soft (dot) com

Register here to receive future CDP Quick Tips by email.

Virtualization is obviously great business for hosting and cloud service providers enabling new capabilities in measuring service levels, provisioning, and high availability.  The big surprise has been that the more we use virtualization the more we have storage performance issues.

Virtualization is all about sharing and making the most of hardware resources.  Let’s face it we would all like each application to have its own dedicated physical server, but when it comes down to it it’s just a matter of cost and manageability, and all we are trying to do is squeeze as much out of our hardware and floor space as possible.

This all sounds feasible with quad-cores becoming common place, memory perhaps getting exponentially less expensive, and 1 Terabyte SATA drives being found in the bargain bin.  The big challenge is that the storage has not really gotten all that much faster in the last two decades.

The way to check-off backups from the list of virtualization storage problems is to use asynchronous-replication type backup software.  What asynchronous replication does is drastically reduce the I/O requests involves in a backup operation.  Drastically like taking 15 minutes instead of 15 hours.  This works by using special device drivers to know ahead of time what parts of your file system or virtual disks have changed since your last scheduled backup operation.  By doing this, only the data that actually changed is read form the disk, and the rest is ignored,  since on average only a fraction of a virtual machine’s data changes on an hourly or even daily basis.  This means very little disk I/O is done during the backup operation.

You can learn more about asynchronous replication products at http://www.r1soft.com

***All attendees will automatically be entered into our Continuous Giveaway for a chance to win an HD Flip Camcorder or an Apple iPad.

Topic: How to Make Money with R1Soft’s CDP Standard Edition
Date: Thursday April 8, 2010
Time: 11:30 AM – 12:30 PM CST
Hosted by: R1Soft

Sign up now to learn about CDP Standard Edition and how it can help you generate additional revenue by offering customer managed backup services.

Topic: Best Linux Server Management Practices
Date: Wednesday April 28, 2010
Time: 11:30 AM – 12:30 AM CST
Hosted by: rackAID

Sign up now for R1Soft’s latest Webcast series to get the most out of your servers and avoid a backup nightmare. Presented by rackAID, the 45-minute Webcast will take an in-depth look at the following topics:

• Backup and Recovery
• Managed Hosting
• Server Management
• Server Monitoring
• IT Security Solutions

More Upcoming R1Soft Events

Here’s a quick tip from the Head GEEK to help you backup MS SQL Server databases using CDP Enterprise Edition.

R1Soft CDP Enterprise Edition for Windows uses MS Volume Shadow Copy Service (VSS) to protect your Windows server and applications like MS SQL Server databases.

1. When protecting MS SQL Server with CDP Enterprise Edition make sure the “SQL Writer Service” is Enabled!  This service must be running when CDP Enterprise Windows Agent requests a backup using Volume Shadow Copy Service (VSS).  The VSS Writer allows the Windows CDP Agent to make a stable asynchronous replica of your running MS SQL Server databases.

2. You can restore individual MS SQL Server databases while SQL Server is running with two easy steps.

To restore a MS SQL Server database:

• Restore the database data and log files using CDP Enterprise Edition file restore

• Attach the restored data and log files to SQL Server using SQL Server Management Studio

Documentation for this tip including step-by-step screenshots is here.

If you would like to receive future CDP Quick Tips by email you can register here.
Have questions about our High-Performance CDP products? Just email headgeek (at) r1soft (dot) com .

Everywhere I look it seems I see “cloud computing“.  Amongst insiders in the hosting industry “the cloud,” has become an inside joke.  I think we all know the word “cloud” is almost meaningless.

And while the word “cloud” may be meaningless it is giving the hosting industry a fresh coat of paint and much needed hype, boosting the sales of hosted services.  Sounds like a celebration for the hosting industry.  It has grown-up.  Ready for prime time.

One thing for sure, the word “cloud” has done wonders to get the investment community excited about hosting and this appears to have improved the valuations of almost any businesses that sells products or services “in the cloud”.  Who can complain about that? …especially if you are a shareholder of one of those companies.

I am going to attempt to define “cloud computing”.  Not as it should be but instead as it is actually used.  You may not like my answer.  I appreciate your feedback.  I’m not sure if it’s even possible to define cloud computing definitively.  I am going to attempt it.

Perception of Cloud Computing

When you think of Cloud Computing you likely came up with one or more of these ideas:

• Utility
• Self-healing
• Grid
• Elastic
• Scalable
• Cluster
• Virtualization
• Maybe brands like: EC2 / S3 / Nirvanix / GoGrid / VPS.net / RackSpace
  

Cloud Computing Defined – Here You Go

Cloud computing is any hosted service over the Internet.

Cloud services typically fit into one of these boxes:

1. Software as a Service examples: Salesforce.com, WordPress.com, eClinicalWorks
2. Storage as a Service examples: Nirvanix, Amazon S3
3. Platform as a Service examples: Amazon EC2, Rackspace Cloud, VPS.net, GoGrid

What Cloud Computing May or May Not Be

Cloud computing May or May Not Be any combination of these:

• Reliable
• Self-Healing (fail-over)
• Scalable
• Virtualized
• Use a Grid
• Elastic
• Clustered

That means when you are shopping for a service “in the cloud,” you need to do your homework.  Do not just assume because it is labeled a “cloud offering” that it has some magical powers.

There are cloud services that have fail-over capabilities, are very reliable, and very scalable.  That does not mean they all are.

A Brief History of the Cloud

I think it is easier to see through the word cloud if we just use a little bit of history.

Virtual Hosting

It seems we started out with this idea of virtual hosting.  Sharing an expensive Internet server among multiple users or customers as a way to sell needed hosted services to people that could not justify a dedicated Internet server.  We used the Apache web server, MS IIS, PHP, ASP, and Bind DNS to cram as many web sites and web applications on a server as possible.

Management Software for Shared Services

Then all of a sudden people realized that sharing their computing resources with strangers can be a problem for three obvious reasons:

1. Management (mostly a service provider problem)
2. Security
3. Sharing limited hardware resources (CPU, memory, disk)

Soon there were software applications to manage the virtually hosted web sites and they came from vendors like CPanel and Plesk.  They solved the management issue and helped some with security.

Virtual Private Servers

Then a new platform for hosting services came along called a Virtual Private Server from vendors like Sphera and Parallels Virtuozzo.  These products added more solid security to the mix and helped a little with the sharing of hardware resources.

Virtualization

Enter Citrix XenServer.  Xen is the dominant platform for services hosted in the cloud that use virtualization.  It solves the security problem, the hardware sharing problem and it does it all with pretty darn good performance*.   Xen probably does not really solve the manageability problem yet… that’s OK… the hosting providers are solving that on their own.

If you have a service in the cloud that uses a virtual machine or virtual private server, chances are it’s on the Xen Virtualization platform.

What about VMWare, I thought VMWare was virtualization?

Yes VMWare was probably the first to solve the virtualization problem and gain mass market share.  VMWare solves all three problems: manageability, security, and sharing hardware.  VMWare did not find its way into the cloud so much.  The reason is that VMWare is not priced in a way that cloud, I mean hosted service providers can make a profit after paying for licenses of VMWare.  Of course there are some providers using it, some even extensively… true.  Overall use of VMWare in the cloud is very small in my humble opinion and will likely continue to remain so as long as Xen is provided basically free of charge. 

* I say Xen has good performance with the caveat that so far, sharing hardware has some pitfalls no matter how you do it… that is for another post.

Summary

Since cloud computing is just a hosted service delivered over the Internet most people in some way are already a customer of some type of cloud service.  Have a web site hosted?  You already use the cloud.  Have a blog account on Wordpress.com?  Use salesforce.com? You have a cloud service.

So while “the cloud” is nothing new, there are some new developments in the technology used to deliver hosted services.  For example Xen has boosted the security and hardware sharing capabilities of cloud services.  R1Soft CDP is the #1 vendor of backup software for data in the cloud.

Time and time again at R1Soft we see disasters that could easily be avoided with some basic reporting and attention to the CDP System after it is deployed.

How To Monitor your CDP Enterprise Installation

If Nothing Else Visit the CDP Task History Screen periodically
http://wiki.r1soft.com/x/twAY

If you do nothing else at least periodically visit the global task history screen on your cdp server.

Schedule Email Reporting
http://wiki.r1soft.com/display/R1D/Scheduling+Email+Reporting

You can create an email report for your entire CDP Server or for just your agents.  You can select to show all log messages of level WARN or higher in the report.

Configure Remote Syslog on Larger Deployments – http://wiki.r1soft.com/display/R1D/Remote+Syslog+Settings

Use a syslog server to receive all log messages from the CDP Server.  Once in syslog the CDP Server messages can be treated like any other log file you probably already monitor.  You can use your method of choice for monitoring: scripts, splunk, a database, nagios script etc.

Windows syslog Server
http://www.kiwisyslog.com/kiwi-syslog-server-overview/

Linux Remote syslog Setup
http://www.aboutdebian.com/syslog.htm
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-General_System_Tuning-syslog_tuning_tips.html

Windows Event Log
http://wiki.r1soft.com/display/R1D/NT+Event+Log+Settings

Examples of why you Need to Monitor your CDP Enterprise Installation

We Do See Hardware Go Bad Sometimes

A few months ago we had a customer with failed restores.  Data on the disk was so corrupt that on one agent the 4 byte magic number at the beginning of the Disk Safe .data file written only once when the disk safe is created was corrupt.    In this case every task on the CDP Server had warnings or errors and most were failing with nasty errors.  They went unnoticed for 1 month until a restore was needed.  That and other warnings from the operating system of a serious problem with disk storage on their CDP Server.

All could have been fixed well before a restore was needed.

Like All Software R1Soft CDP Has Errors and Often there are Warning Signs

Linux Agent s with version numbers lower than 1.67.0 (CDP Release 2.19) as reported in our release notes in some cases can fail to properly identify deltas and read blocks correctly on disks or volumes larger than 2 TB.  The issue was fixed in Linux Agent 1.67 in November.  The symptoms include complete failure to add the device/partition to the CDP Server OR literally thousands of warnings during the backup about corrupt or missing blocks.
http://wiki.r1soft.com/x/W5Jy

The thousands of errors during each backup this issue can trigger are part of a validation done during backup to make sure none of the blocks allocated by the file system are missing from CDP replication.  If blocks are missing it can be an issue with the software, hardware, or a corrupt file system.

The File System on Your Server (where CDP Agent runs) May be Corrupt – In Some Cases CDP Can Detect and Warn about This

It’s actually not that uncommon for Linux and Windows file systems to become corrupt to one degree or the other especially on multi-tenant systems that rarely if ever get taken down for a periodic fsck or checkdisk of the file system (cuz we all love a 24 hour fsck… right).

While rare what can happen is some file system data structures on disk become corrupted.  Could be hardware issue, could simply be caused by a hard reset of the server.  These data structures in some cases are also resident in memory on the operating system.  This allows the server to pretty much function mostly correctly though there are often clues from the kernel in /var/log/messages.  The real surprise is when you reboot this server when you find out the file system corrupted beyond repair.

In some cases (not all) the CDP Server will give warnings signs.  Unable to read block XXXXX I/O error (you have bad sectors or some storage failure).  Warning blocks are corrupt which means they are allocated by the file system but the CDP Software never detected they were written to.  This typically means file system corruption and the file systems block allocation maps have issues.

Other good reasons to Monitor if these are not enough:

• A firewall may have changed blocking access to the agent from the CDP Server.
• There could be damage to a Disk Safe from a CDP Server crash or power loss .
• You could have run out of disk space on your CDP Server
• Backups may be working but MySQL may not be locked and flushed during the backup because the agent is unable to connect to MySQL

Improvements in Reporting Based On Customer Feedback

CDP 2.20 – Tasks Set to Warn State of MySQL Lock and Flush Fails

Task Alerts – The next major release of CDP has alerts.  Alerts  are added to a task if it needs attention for any reason.  Alerts contain a very simple message of basically what went wrong.  Inside of an alert are all the relevant details associated with the issue to troubleshoot it.
http://www.r1soft.com/linux-cdp/cdp-standard-edition/screen-shots/

Dashboard – Included in the next major CDP release is a dashboard.  This can at a glance give you indicators that may require further investigation.  Its very cool but no silver bullet.  The dashboard must be looked at and in a large CDP Enterprise installation it’s a challenge to convey everything important on a dashboard.

Protection from Full Disks – CDP 3 with a Beta standard edition available for download right now has built in soft and hard disk quotas.  By default you will receive warnings if the disk/volume where a disk safe is stored has less than 20% free space.  With less than 10% free space backups will fail.

Localization – We have localized everything in the next CDP major version include most error messages.  What good are errors if you can’t understand them?  I agree.

Here’s a quick tip from the Head GEEK to help you balance backup compression so it doesn’t adversely affect backup time and performance.

R1Soft CDP Enterprise Edition for Windows and Linux uses two techniques to reduce the size of data archived in your recovery points.

1. The R1Soft Virtual Full Backup process only stores unique block-level deltas (changes since your last backup) one time across all of your archived recovery points.

2. Compression can be enabled to reduce the size of the deltas that are stored.

CDP Enterprise Edition uses the zlib software library for compressing deltas in your recovery points.  In addition to the savings that Virtual Full Backups provide, compression that can further reduce the backup data storage by as much as 50% (2:1 compression).  Compression does have a cost.  It slows down the backup and restore transfer rates.  Compression also consumes CPU cycles.  Compression is performed where the CDP Agent is installed before data is transmitted over the network so the CPU usage impacts your production servers.

Here is how to best use compression:
• For fastest backup and restore speeds use “No Compression”

• Compression is best for performing CDP backups over slower Wide-Area-Network links like 10 Mbit Ethernet or T-1s.

• If you do need compression, level 1 is usually best.  In most environments it will typically provide a 50% reduction in storage space.

• Try compression levels higher than 1 only if your production server has plenty of CPU cycles to spare and backup and restore performance is not as important as saving storage space on your CDP Server.

Tip KB article is here: http://wiki.r1soft.com/display/R1D/Disk+Safe+Compression

If you would like to receive future CDP Quick Tips by email you can register here.

Have questions about our High-Performance CDP products? Just email headgeek (at) r1soft (dot) com

Here’s a quick tip from the Head GEEK to help you ensure safe, fast online backup and zero business interruption for your MySQL database servers.

When it comes to backups databases require special handling, and MySQL is no exception. There are a variety of MySQL backup methods available and they all work differently.  The R1Soft CDP for MySQL Add-on integrates Continuous Data Protection® with an online hot backup (snapshot) of MySQL databases to provide fast, efficient and safe MySQL backups.  Both MyISAM and InnoDB storage engines require that tables be locked and flushed before a backup operation can take place. The R1Soft MySQL Add-on ensures that this happens and  carefully coordinates it with the point-in-time file system snapshot.

If you use CDP to backup MySQL and don’t use the R1Soft MySQL Add-on then the table lock and flush operation won’t take place – with the result that your MySQL backup may very likely be corrupt!

Tip KB article can be found here.

If you would like to receive future CDP Quick Tips by email you can register here.

Have questions about our High-Performance CDP products? Just email headgeek (at) r1soft (dot) com

If you are signed up to receive R1Soft emails then you may have already seen our new CDP Quick Tips. We will continue to send out quick tip emails, but will also start posting them on the blog as well. CDP Quick Tips offer insightful information on best practices for data protection using R1Soft CDP products. If you would like to receive CDP Quick Tips by email you can register here.

CDP Quick Tip #1:

The R1Soft CDP Server archives virtual full backups on disk.  Once the initial replica is complete it snapshots changes (block level deltas), organizes them into recovery points, and stores them in the R1Soft Disk Safe data files. Some of the block level deltas that make up the recovery points may well be stored for years, because they haven’t changed.  Because this data is stored for long periods of time it becomes susceptible to file system fragmentation.  This typically gets worse over time and can cause the performance of the CDP Server to erode.

To ensure that your CDP Servers continue to perform at high speed, I recommend that you defragment them on a regular basis.

Tip KB article can be found here.

Have questions about our High-Performance CDP products? Just email headgeek (at) r1soft (dot) com

Following up with my last post about why the CDP MySQL Add-On is so important to getting a good backup of your MySQL database I have been asked a lot: “What good is the point-in-time snapshot anyways if it’s not good enough for MySQL? What about XYZ application?”

Beauty of the Snapshot is in the Eye of the Beholder

I think it’s hard to talk about volume snapshots without talking about who you are and what applications you use.  If you are a database admin you have one view, a Linux server admin a different view, and a Windows server admin yet a third view.

Windows Server Admins

You know you need some way to get a backup copy of open files as Windows enforces file locking and windows application typically lock files for reading.  When you shop for backup software you are used to making sure it has some kind of support for backing up open files.  Before MS Volume Shadow Copy you may have purchased expensive third party tools like OTM (open transaction manager) to solve this problem with your backups in the past.

Linux Server Admins

You probably never thought much about snapshots or consistency of files during backup.  On Linux the root user can read any file it wants and at any time regardless if the file is being written to or not.  You can even read locked files. Basically you are not protected from yourself unlike Windows where file locking really locks files. 

Database Administrators

You know your database needs more than just a file copy when it comes to backup.  You perform a scheduled database dump at a minimum, you likely purchase add-ons from your Windows/Linux file backup vendor that supports your database (e.g. R1Soft).  You may even purchase a specialized commercial database backup tool made just for your database.

What is a point-in-time File System Snapshot (aka Open File backup)?

Windows

The CDP Agent performs an online point-in-time snapshot using the Microsoft Volume Shadow Copy Service (VSS).

For the file system this means:

1. CDP Agent requests a snapshot from the VSS service
2. Volume Shadow Copy first notifies all applications that have asked to be notified there is an impending backup operation coming and to write critical data to disk.
3. VSS locks and flushes the file system.  Writes cached in memory and not yet written to disk are now flushed through to disk storage.
4. The file system is unlocked
5. Applications are notified the snapshot is complete and to continue writing as normal
6. CDP Agent reads backup data from the snapshot

Changes to the file system (writes to a file, new files, etc) that happen after the snapshot don’t appear in the backup.

For applications this means:

On Windows we have the MS VSS Service which provides just about all the snapshot features you could ever want.  (Linux really is lacking here).  And while we love to hate VSS when it does not work… it is very powerful and credit is due here.

Any application that decided to support VSS can inform VSS that it should be informed when there is a backup operation so it can write un-saved changes in memory to disk. MS VSS calls this a VSS writer.

So the question to ask on Windows is: Does my application integrate with Microsoft VSS.  Microsoft calls this a VSS Writer.

Linux

Linux has no Volume Shadow Copy Service and no snapshot support in the popular file systems (ext2/3/4, reiserfs, XFS, etc).  To bring VSS type volume snapshots to the Linux world R1Soft developed a special device driver (kernel module) for Linux that can perform a point-in-time snapshot of any Linux block device. Now that’s not all the CDP Linux device driver does… it has some magic that perform asynchronous replication… making backup windows very short… that is another matter.

For the file system this means:

1. The CDP Agent calls a ioctl call on the CDP driver to request a snapshot
2. If the MySQL add-on is installed MySQL is locked and flushed here
3. CDP driver locks and flushes the file system.  Writes cached in memory and not yet written to disk are now flushed through to disk storage.
4. If the MySQL add-on is installed MySQL is unlocked here
5. CDP driver unlocks the file system
6. CDP Agent reads backup data from the snapshot

Changes to the file system (writes to a file, new files, etc) that happen after the snapshot don’t appear in the backup.

So the question to ask on Linux is: Is my application safe with or without just a point-in-time snapshot.  And does my Linux application need special treatment during a backup.

Linux –  backup applications that do NO Snapshot like tar, rsync, bacula, amanda are they safe?

No, tar, rsync, bacula, amanda and just about every other Linux backup application* besides R1Soft CDP are reading from files while other applications are writing to them.  They are actually WORSE OFF.  There is no point-in-time snapshot so you get a mix match of changes to the file system in your backup and there is no protection at all to backing up open files.  If you use these utilities to backup Linux you are almost guaranteeing corrupt files!  Sorry, the truth is ugly I know… best we are aware.

What does this Snapshot mean for applications?

It depends on the application and how it does disk I/O and what is doing. Here is a summary of some popular applications that I have done the homework for you on:

Windows

Life is easy as most critical apps support VSS and MS did all the hard work.

Linux

R1Soft has made your life much better by giving you a point-in-time snapshot… life is still complicated because there is no VSS and few application developers take time to solve the consistent backup problem.

*Acronis True Image and Linux backup applications that use LVM to perform snapshots assuming all your data is on LVM Logical Volumes are also capable of performing point-in-time snapshots.  Zamanda can use LVM snapshots and this is good if all your data lives on LVM and you set aside special snapshot storage ahead of time.

**PostgreSQL is safe to copy at any time because of the way its magnificent write journal works. See http://www.r1soft.com/linux-cdp/cdp-server-20/postgresql/ for more details.

***Postfix mail spool files in my opinion are safe with only a file system snapshot with the caveat that mail spool files may have partially written or deleted messages.