The Storage for distributed collaborations typically consists of an array of distinct storage elements—each within separate administrative domains—that may be connected together with a common Grid transfer and security mechanism. Data transfers are organized around files, which are copied between sites. A Storage Cloud, the other hand, can be distributed in nature, will look the same from all access locations, and defines a single administrative domain for all its data. This is the L-Store File System model. Data transfers within a storage cloud can, like a typical file system, be organized around data blocks arbitrarily located within the cloud.
Logistical Networking
The power and flexibility of L-Store can be traced to Logistical Networking (LN), a distributed storage architecture designed explicitly to offer a scalable solution to problems of data logistics in distributed computing in the WAN context. Many of the advantages of LN derive from the way in which it handles file metadata. Conventional file systems operate on a file abstraction that consists of two kinds of stored information: the actual data that is the contents of the file; and structural metadata, which we also call a data mapping. When a file is written to a file system, its contents are divided into pieces (blocks) and each block is written and stored separately. The map to where each block is stored and other state information is the structural metadata. In UNIX, this metadata is called the inode. Typically, the user does not have access to this metadata.
ExNode
LN uses a generalized abstraction of the inode, a container called the exNode. Information about a distributed file’s data mappings is placed in the exNode. In addition to the file’s structural metadata, arbitrary non-structural metadata, called attributes, can be stored in the exNode. The generality of this abstraction allows us to combine storage resources from many sources, implemented by many different technologies. Files stored in this distributed mix of hardware, along with their metadata, can be manipulated by users through widely used interfaces with familiar semantics.
Internet Backplane Protocol
The fundamental unit of storage in LN is the depot. A depot can be a single disk or a collection of disks in a server. In LN, the pieces or blocks of a file are called allocations.
As currently implemented in L-Store, exNodes use one basic storage access protocol—the Internet Backplane Protocol (IBP). IBP provides a generic, best-effort service that allocates, reads, writes and manages allocations on network-addressable storage (the depots). The exNode data mappings capture the way in which the (possibly replicated) segments of a data extent are mapped to IBP allocations. By factoring the data movement and storage and data-mapping aspects of the file abstraction, the LN architecture exposes the structural metadata in a form that can be safely and directly managed and manipulated by client processes and by services acting on behalf of the client. IBP allows for third party transfer of data between depots relieving users of the burden of moving data around the Internet, as other middleware tools can do this on their behalf.
The L-Store software stack implements a complete virtual file system. L-Store uses IBP as the underlying abstraction of distributed storage. It manages, monitors, and maintains data distributed to IBP depots, and supports RAID-like mirroring and striping of data for performance and fault tolerance.
Lifecycle Management
L-Store supports the complete lifecycle management of hardware. No downtime is required to add disk storage, which can be added on the fly with the space becoming immediately available. As hardware is added, allocations can be redistributed to maintain uniformity in data distribution. Hardware can be retired, resulting in the automatic migration of data off the retiring hardware. IBP supports heterogeneous disk sizes and storage hardware. As a result, L-Store can grow based on demand, using the best technology. This has become a routine and common occurrence in the multi-petabyte L-Store storage element used at the Vanderbilt CMS Tier2 center.
Interacting with the L-Store File System
There are two main methods for interacting with L-Store: command line tools, and a FUSE based disk mount. The Logistical Input/Output (LIO) command line tools provide the most complete and efficient way to access the L-Store data. These tools are designed to replicate familiar Linux commands such as cp, du, find, fsck, ln, ls, mkdir, mv, rm, rmdir, and touch. A person with a reasonable familiarity of these Linux commands can easily transition to using the LIO command line tools. The LIO tools also provide commands for L-Store specific tasks such getting and setting file attributes, extending the lifetime of a file, directing output to standard out or input to standard in, or inspecting the health and physical location of files. These tools are easily incorporated into shell scripts and the like allowing for straightforward access, manipulation, and administration of files in L-Store.
The second access method is even more user friendly and transparent than the LIO tools. L-Store takes advantage of a kernel module that is readily available on most Linux distributions and Mac systems, the File system in Userspace (FUSE) kernel module (http://fuse.sourceforge.net/). L-Store provides a FUSE-based interface called the Logistical File system (LFS). With minimal configuration, and using a single command, LFS can be mounted on an investigators machine as if it were an external hard drive. This mount point allows the user to carry out analysis as if the data was located on his or her local machine.
Instances of L-Store have been in production use for over three years at the Advanced Computing Center for Research and Education (ACCRE), and it is the ACCRE team who are the primary developers of L-Store software. The CMS repository at ACCRE currently stores over a petabyte of CMS data, and it has been pushed hard by CMS users. There are, however, significant development efforts that are being undertaken to assure that L-Store meets the evolving needs of CMS.