Posts tagged as:
It’s a rather dark and snowy Tuesday in Hopkinton today but, despite the craptastic weather, there’s some good news floating about. EMC’s Cloud Infrastructure Group has released Atmos version 1.3 which is a significant overhaul of the Atmos code previously released. So, what’s new, what’s cool, and why even bother?
{ 5 comments }
As many of you may know, I work for EMC’s Cloud Infrastructure Group as part of the Atmos solution team. In this role, I’ve been blessed with getting a closer look at where the future of cloud storage is going as well as some of the drivers that will get it there. In this post, I’d like to talk a bit about policy and how this will shape the future of storage. I’m going to keep this as abstracted from product as possible, but where appropriate, I’ll try to show you how products are implementing this technology TODAY.
What is Policy?
By definition, policy is “[an] action or procedure conforming to or considered with reference to prudence or expediency” (dictionary.com for that definition). When viewed in the context of storage systems and management, policy, then, is the actions (scripted or otherwise) that influence data to provide for retrieval, performance, or manipulation by systems. In other words, policy is an engine that manages data from start to finish. Why this is important requires us to look at what the typical management stack looks like today.
Data is created by users accessing programs that are tied to physical and virtual resources. This generated data is then processed and stored by the programs and their underlying storage I/O layers (LVMs, hypervisor I/O stacks, etc.) onto some sort of storage device (SAN, NAS, DAS, etc.) where it sits until next access. In essence, once data is created it is considered to be “at rest” until it is next accessed (if ever). Within this data generation and storage continuum, the process is fundementally simple as generated data is put directly to storage. However, if the data continues to sit in the same place endlessly, it’s typically inefficient to retrieve and access. Managing this data was typically a manual process where data, LUNs, and their topologies had to be moved around using array or host-based tools to provide better “fit” for data at rest or data accesses for performance. This is where policy steps in.
Policy uses hooks into data (also known as metadata) in order to enact controls. Please see this post for more detailed explanation of metadata.
Why use Policies?
If the previous example shows anything, it’s that the management of data is fundementally…well, boring and manual. Policy provides a method of controlling the stack of data ingest AND data management while allowing business to continue to generate, retrieve, and manipulate data. For example, a simple policy that could be enacted against data could be as follows:
if data < 14 days old, store on EFD drives, LUN 11; if > 14 days old, store on SATA drives, LUN 33
Obviously, that’s a high-level abstraction of what the actual process for data control would look like but drives the point home. What used to be a manual LUN migration policy to “performance” or “store” data now is set based on a logical control structure that can be automagically enacted on the storage system itself. A working example of this type of policy can be seen in the tiering provided by Compellent and EMC’s FAST systems for storage management. Pretty cool, huh?
An alternative method of control that isn’t necessary tied to the storage array is the recent introduction of VMware’s Storage DRS (Dynamic Resource Scheduling) which is enacted against the storage I/O stack of VMware’s vSphere hypervisor.
The Future of Policy
Obviously, my examples are very simplistic in nature but hopefully, they make the policy technology somewhat more accessible. As far as policy futures are concerned, this is where storage technologies (and even host process management) will be going. In the future, simple policy creation and enforcement will be a necessary part of storage pool creation and integration as well as the ongoing maintenance and support of storage arrays.
As always, feedback is welcome!
edit: 9/21/09: removed a mis-aligned reference to Atmos storage policy.
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=7193e1ae-9be7-481e-bb57-34d1f58ad77e)
{ 9 comments }
One of the principle points of contention within the cloud storage space is what to do with metadata (literally: “data about data”). There are several schools of thought regarding how metadata should be presented, protected and optimized but for the sake of this post, I’ll just tackle 2 of them. These two thought processes are: (a) metadata appended directly to the objects that they represent and (b) metadata separated from the object and stored separately in a database or other type of referencable system object. Let’s unpack some basic advantages/disadvantages of each approach.
Metadata wrapped directly around the object
Image by Matthew Bietz via Flickr
I’m going to refer to this first world view as the “bacon-wrapped scallops” worldview. Objects (the scallop) are wrapped in their respective metadata references (the bacon). The benefit here is that the metadata becomes an intrinsic part of the object that’s being referenced. Replication policies, etc. can be directly enacted on an object without much cause or care for concern. Additionally, any loss of metadata reference affects a singular object only, not globally, thus restraining chain failures across a filesystem.
Issues with this approach are fundamentally tied to portability, corruption and performance. Portability is a double-edged sword. It’s great when you want to make sure that the object can exist in multiple locations but since the metadata is bound to an object, locality of reference can be skewed. In other words, having an extant meta database provides simple records updating for replication and locality whereas bound meta requires updates to EACH replica object. This also ties into performance as this update process across XXX of objects can take time as it’s a hunt operation within a matrix of (potentially) billions of files. Corruption, finally, can potentially wreak havoc in a wrapped model because each object meta can be corrupted and, when sync’ed to replicas can pass on corruption.
Metadata as distinctly separate entity
Scallops with Creamed Corn, Asparagus, Pearl Onions and Coffee Cocoa Chile Butter
If metadata being wrapped to the object is the “bacon-wrapped scallops” approach, then the concept of keep metadata separate from the object is, well, “Scallops with Creamed Corn, Asparagus, Pearl Onions, and Coffee Cocoa Chile Butter.” (don’t laugh, you can find this recipe here). Obviously, this is a more complicated recipe than just “bacon-wrapped scallops” but it does prove a certain point: separating metadata can have an obvious benefit of “improving” the base characterization of the underlying object while allowing for methods of portability (exemplified by the recipe above) and for customization that exceed meta-wrapped objects. Advantages include the ability to provide UID reference points in an extant database that is subject to its own protection schemas (either by replication or otherwise), performancing (meta references need only be updated to the db by design, not to the object(s) present), and corruption prevention (object corruption occurs independent of meta and vice-versa).
Disadvantages would still be present as you now have to design for an extant meta database(s) to equalize performance across nodes (as noted in my master/slave node topology here). Additionally, a protection mechanism for these databases could have performance implications (especially in synchronous, off-box schemas) as a vast number of records (potentially, again, billions) would have to be appended. Another potential double-edge sword is the need to have some level of metadatabase replication in place to other nodes within the cloud storage platform in order to enforce meta-driven policies. Since the reference model is based on object UID in the database, these databases have to be persistent through the entire platform, potentially driving up storage capacity utilization.
Hopefully, this makes sense and I’d appreciate any feedback that you might have!
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=53e74041-5f44-4400-8b01-ba7331539df8)
{ 24 comments }
