Comments on: Micro-burst: Metadata http://flickerdown.com/2009/08/micro-burst-metadata/ a Blog dedicated to storage and technology Wed, 03 Mar 2010 13:51:46 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Towards a Private Cloud Architecture » privatecloud.com http://flickerdown.com/2009/08/micro-burst-metadata/comment-page-1/#comment-374 Towards a Private Cloud Architecture » privatecloud.com Tue, 03 Nov 2009 16:30:43 +0000 http://flickerdown.com/?p=664#comment-374 [...] leaves us with the legacy-free information abstraction—objects wrapped in rich metadata.  Note: the visuals offered up by Dave Graham on this topic are not to be [...] [...] leaves us with the legacy-free information abstraction—objects wrapped in rich metadata.  Note: the visuals offered up by Dave Graham on this topic are not to be [...]

]]> By: Why Policy is the future of storage — Dave Graham's Weblog http://flickerdown.com/2009/08/micro-burst-metadata/comment-page-1/#comment-345 Why Policy is the future of storage — Dave Graham's Weblog Sun, 20 Sep 2009 14:58:54 +0000 http://flickerdown.com/?p=664#comment-345 [...] uses hooks into data (also known as metadata) in order to enact controls.  Please see this post for more detailed explanation of [...] [...] uses hooks into data (also known as metadata) in order to enact controls.  Please see this post for more detailed explanation of [...]

]]> By: Why Policy is the future of storage — Dave Graham's Weblog http://flickerdown.com/2009/08/micro-burst-metadata/comment-page-1/#comment-344 Why Policy is the future of storage — Dave Graham's Weblog Sun, 20 Sep 2009 14:58:54 +0000 http://flickerdown.com/?p=664#comment-344 [...] uses hooks into data (also known as metadata) in order to enact controls.  Please see this post for more detailed explanation of [...] [...] uses hooks into data (also known as metadata) in order to enact controls.  Please see this post for more detailed explanation of [...]

]]> By: Derek Gascon http://flickerdown.com/2009/08/micro-burst-metadata/comment-page-1/#comment-407 Derek Gascon Wed, 26 Aug 2009 23:59:16 +0000 http://flickerdown.com/?p=664#comment-407 I'm a little late to this conversation, but it's important and one I sure hope continues. Metadata and storage is an area I've been involved in since the late 90's and have developed product strategy for content management and index/search software as well. The value of metadata was little understood in storage back in the old days and it was a challenge to get the concept across. The best way to consider metadata is to use a metaphor we're all familiar with, the library. In fact, it's a good way to really think about storage in general. In the library books are stored in rows and on shelfs with the title showing and based on a certain taxonomy (whole science around this library stuff ya know). This is analogous to the file system using the folder, sub-folder and file name. Now, in the library you have the card catalog that contains basic information (metadata) to help you locate a particular book. Each book contains that basic information as well as an index (additional metadata) that describes in more detail what's in the book. This is an example of the hybrid model you suggest. Let's say there's a flood and the card catalog is lost, but the books remain. The librarian can rebuild the card catalog from the information contained within the book. <br><br>In the digital world ECM, records management and other applications store the most important metadata in a database and the file is stored in some folder. The context of that file in some folder is lost if that database is lost or disappears after a period of years. However, by storing more descriptive metadata along with the file allows that context to persist over time without worrying about losing the ability to easily determine the value of a file. It's what I like to call Content in Context. The question of how much metadata to store in an object becomes a question for the organization and its requirements for information management. There are a number of industry standards for metadata that can be used for specific types of content such as the NBII Biological Metadata Standard, the Content Standard for Geospatial Metadata and Dublin Core (there are many more). Perhaps only a subset of the metadata standard is required or maybe all of it. That's a level of flexibility information/records managers should have. A lot of content is going to be kept for decades and keeping that context alive is critical. The hybrid model makes a lot of sense, at this point in time and I expect new applications will emerge that take advantage of object stores allowing dynamic views/organization of content/files. <br><br>Derek Gascon<br>VP Marketing, Caringo, Inc. I'm a little late to this conversation, but it's important and one I sure hope continues. Metadata and storage is an area I've been involved in since the late 90's and have developed product strategy for content management and index/search software as well. The value of metadata was little understood in storage back in the old days and it was a challenge to get the concept across. The best way to consider metadata is to use a metaphor we're all familiar with, the library. In fact, it's a good way to really think about storage in general. In the library books are stored in rows and on shelfs with the title showing and based on a certain taxonomy (whole science around this library stuff ya know). This is analogous to the file system using the folder, sub-folder and file name. Now, in the library you have the card catalog that contains basic information (metadata) to help you locate a particular book. Each book contains that basic information as well as an index (additional metadata) that describes in more detail what's in the book. This is an example of the hybrid model you suggest. Let's say there's a flood and the card catalog is lost, but the books remain. The librarian can rebuild the card catalog from the information contained within the book.

In the digital world ECM, records management and other applications store the most important metadata in a database and the file is stored in some folder. The context of that file in some folder is lost if that database is lost or disappears after a period of years. However, by storing more descriptive metadata along with the file allows that context to persist over time without worrying about losing the ability to easily determine the value of a file. It's what I like to call Content in Context. The question of how much metadata to store in an object becomes a question for the organization and its requirements for information management. There are a number of industry standards for metadata that can be used for specific types of content such as the NBII Biological Metadata Standard, the Content Standard for Geospatial Metadata and Dublin Core (there are many more). Perhaps only a subset of the metadata standard is required or maybe all of it. That's a level of flexibility information/records managers should have. A lot of content is going to be kept for decades and keeping that context alive is critical. The hybrid model makes a lot of sense, at this point in time and I expect new applications will emerge that take advantage of object stores allowing dynamic views/organization of content/files.

Derek Gascon
VP Marketing, Caringo, Inc.

]]> By: Derek Gascon http://flickerdown.com/2009/08/micro-burst-metadata/comment-page-1/#comment-338 Derek Gascon Wed, 26 Aug 2009 18:59:16 +0000 http://flickerdown.com/?p=664#comment-338 I'm a little late to this conversation, but it's important and one I sure hope continues. Metadata and storage is an area I've been involved in since the late 90's and have developed product strategy for content management and index/search software as well. The value of metadata was little understood in storage back in the old days and it was a challenge to get the concept across. The best way to consider metadata is to use a metaphor we're all familiar with, the library. In fact, it's a good way to really think about storage in general. In the library books are stored in rows and on shelfs with the title showing and based on a certain taxonomy (whole science around this library stuff ya know). This is analogous to the file system using the folder, sub-folder and file name. Now, in the library you have the card catalog that contains basic information (metadata) to help you locate a particular book. Each book contains that basic information as well as an index (additional metadata) that describes in more detail what's in the book. This is an example of the hybrid model you suggest. Let's say there's a flood and the card catalog is lost, but the books remain. The librarian can rebuild the card catalog from the information contained within the book. <br><br>In the digital world ECM, records management and other applications store the most important metadata in a database and the file is stored in some folder. The context of that file in some folder is lost if that database is lost or disappears after a period of years. However, by storing more descriptive metadata along with the file allows that context to persist over time without worrying about losing the ability to easily determine the value of a file. It's what I like to call Content in Context. The question of how much metadata to store in an object becomes a question for the organization and its requirements for information management. There are a number of industry standards for metadata that can be used for specific types of content such as the NBII Biological Metadata Standard, the Content Standard for Geospatial Metadata and Dublin Core (there are many more). Perhaps only a subset of the metadata standard is required or maybe all of it. That's a level of flexibility information/records managers should have. A lot of content is going to be kept for decades and keeping that context alive is critical. The hybrid model makes a lot of sense, at this point in time and I expect new applications will emerge that take advantage of object stores allowing dynamic views/organization of content/files. <br><br>Derek Gascon<br>VP Marketing, Caringo, Inc. I'm a little late to this conversation, but it's important and one I sure hope continues. Metadata and storage is an area I've been involved in since the late 90's and have developed product strategy for content management and index/search software as well. The value of metadata was little understood in storage back in the old days and it was a challenge to get the concept across. The best way to consider metadata is to use a metaphor we're all familiar with, the library. In fact, it's a good way to really think about storage in general. In the library books are stored in rows and on shelfs with the title showing and based on a certain taxonomy (whole science around this library stuff ya know). This is analogous to the file system using the folder, sub-folder and file name. Now, in the library you have the card catalog that contains basic information (metadata) to help you locate a particular book. Each book contains that basic information as well as an index (additional metadata) that describes in more detail what's in the book. This is an example of the hybrid model you suggest. Let's say there's a flood and the card catalog is lost, but the books remain. The librarian can rebuild the card catalog from the information contained within the book.

In the digital world ECM, records management and other applications store the most important metadata in a database and the file is stored in some folder. The context of that file in some folder is lost if that database is lost or disappears after a period of years. However, by storing more descriptive metadata along with the file allows that context to persist over time without worrying about losing the ability to easily determine the value of a file. It's what I like to call Content in Context. The question of how much metadata to store in an object becomes a question for the organization and its requirements for information management. There are a number of industry standards for metadata that can be used for specific types of content such as the NBII Biological Metadata Standard, the Content Standard for Geospatial Metadata and Dublin Core (there are many more). Perhaps only a subset of the metadata standard is required or maybe all of it. That's a level of flexibility information/records managers should have. A lot of content is going to be kept for decades and keeping that context alive is critical. The hybrid model makes a lot of sense, at this point in time and I expect new applications will emerge that take advantage of object stores allowing dynamic views/organization of content/files.

Derek Gascon
VP Marketing, Caringo, Inc.

]]> By: dave http://flickerdown.com/2009/08/micro-burst-metadata/comment-page-1/#comment-337 dave Mon, 24 Aug 2009 15:22:01 +0000 http://flickerdown.com/?p=664#comment-337 Thanks for the read, Ian! definitely happy to take a look at what you've written! Thanks for the read, Ian! definitely happy to take a look at what you’ve written!

]]> By: ianhf http://flickerdown.com/2009/08/micro-burst-metadata/comment-page-1/#comment-336 ianhf Sat, 22 Aug 2009 13:11:51 +0000 http://flickerdown.com/?p=664#comment-336 Dave,<br><br>Sorry for the delay, good post and blog commenst!<br><br>I've tried to collate some of my thoughts and comments here :-<br><br><a href="http://grumpystorage.blogspot.com/2009/08/objects-metadata.html" rel="nofollow">http://grumpystorage.blogspot.com/2009/08/objec...</a><br><br>Cheers<br><br>Ian Dave,

Sorry for the delay, good post and blog commenst!

I've tried to collate some of my thoughts and comments here :-

http://grumpystorage.blogspot.com/2009/08/objec...

Cheers

Ian

]]> By: om_nick http://flickerdown.com/2009/08/micro-burst-metadata/comment-page-1/#comment-335 om_nick Fri, 21 Aug 2009 14:15:34 +0000 http://flickerdown.com/?p=664#comment-335 As well as all the other good things (clustered storage stylee) that MatrixStore does search is a biggy. Like you say, we tie into Final Cut Server and other MAM/DAM apps to pull out the important stuff that can be archived with the data. As well as all the other good things (clustered storage stylee) that MatrixStore does search is a biggy. Like you say, we tie into Final Cut Server and other MAM/DAM apps to pull out the important stuff that can be archived with the data.

]]> By: Dave Graham http://flickerdown.com/2009/08/micro-burst-metadata/comment-page-1/#comment-334 Dave Graham Fri, 21 Aug 2009 14:08:21 +0000 http://flickerdown.com/?p=664#comment-334 I think I also missed commenting on one of the principles driving the use of metadata as well (which you and others like @piewords) have mentioned: search! Metadata is absolutely critical for the ongoing control and (dare I say it?) manipulation of content. By giving you hooks to the underlying object, you can now operate against an object in ways that you couldn't before. <br><br>Additionally, you're right in saying that this isn't just a cloud problem. I look at the previous work I've done with SAN-based products and we're so used to looking at data as 01010101 versus a more descriptive method of understanding. I think Tom Maguire made the point earlier that a movie file (for example) is nothing but a file name in a file system until you start describing it and providing characterization against it for your programs. (since I know you do stuff with Final Cut Pro). sorry my thoughts this morning are a little less collected than yesterday but I appreciate the dialogue!<br><br>cheers,<br><br>Dave I think I also missed commenting on one of the principles driving the use of metadata as well (which you and others like @piewords) have mentioned: search! Metadata is absolutely critical for the ongoing control and (dare I say it?) manipulation of content. By giving you hooks to the underlying object, you can now operate against an object in ways that you couldn't before.

Additionally, you're right in saying that this isn't just a cloud problem. I look at the previous work I've done with SAN-based products and we're so used to looking at data as 01010101 versus a more descriptive method of understanding. I think Tom Maguire made the point earlier that a movie file (for example) is nothing but a file name in a file system until you start describing it and providing characterization against it for your programs. (since I know you do stuff with Final Cut Pro). sorry my thoughts this morning are a little less collected than yesterday but I appreciate the dialogue!

cheers,

Dave

]]> By: om_nick http://flickerdown.com/2009/08/micro-burst-metadata/comment-page-1/#comment-333 om_nick Fri, 21 Aug 2009 14:01:47 +0000 http://flickerdown.com/?p=664#comment-333 Not simply a problem for the cloud. For those not wanting to reach for the skies there are storage solutions available now that offer this.<br><br>Glad you tackled Metadata as for us it is becoming as important and the essence itself. Not simply a problem for the cloud. For those not wanting to reach for the skies there are storage solutions available now that offer this.

Glad you tackled Metadata as for us it is becoming as important and the essence itself.

]]>