I have successfully created a stripe across two computers using the GlusterFS install. It was pretty easy to setup and configure though the documentation they have on their site needs a little revision. At this point it’s really written towards someone with a higher level of understanding of both linux and the hardware involved.
Now the setup I have experimented with was just a means for me to see what would happen if I setup 2 machines, one with a 200GB partition and the other with a 60GB partition. After I configured both machines and mounted the GlusterFS clustered/striped file system to /mnt/pool, I had a nice, shiny ~250GB partition. This is cool for 3 reasons.
1. I don’t need the same hardware in both machines to make this setup work.
2. I get approx 100% of both drives being used in the setup. HD1 + HD2.
3. I get a slight speed increase due to the striping, similar to RAID 0. When data is written or read from the drives, I have 2 sources instead of one. So my maximum theoretical bandwidth is no longer limited to hard drive transfer rates, but to the 2 system’s drives or the network, whichever comes first.
I am testing these out on a slower network setup, 100Mb/s bandwidth, but once I have a more advanced setup I want to move to something a little more robust, 1000Mb/s.
My setup right now has one MAJOR flaw in it. If one drive goes down (drive, system, network) then the whole /mnt/pool file system is unavailable. Again, think Raid 0. For me this would only be useful if I had a bunch of drives that I just wanted to use as a faster way to serve unimportant data.
Since all the data I have is important, that leaves me with 2 options. Simple mirroring across multiple drives (Raid 1) or what I’m going to try in my next step: File replication over striping. Think more like Raid 5. Benefits will be the following:
1. Redundant data. Data is mirrored across multiple drives to allow redundancy and a speed increase.
2. Speed increase. Since data is stored on more than one drive, the machines read from multiple locations and faster read times can be achieved.
3. Storage space. While not as impressive as Raid 0 on it’s use of space, Raid 5 uses a minimum of 3 drives and stores like this.
Drive one: 1/2/3
Drive two: 2/3/1
Drive three: 3/1/2
Data is figured being “Size * (Number of disks – 1) / number of disks). So assuming these are three 1TB drives, it would look like this: 3,000,000,000 * (3 – 1 = 2) / 3 = 2,000,000,000. So you get storage space equal to ~66% of the total drive space in the above example. Three 1TB drives will give you 2TB of usable space. Three 100GB drives give you 200GB usable. Assuming you use 5 drives…5,000,000,000 * (5-1=4)/5 = 4TB. 3 gives you 2, 5 gives you 4, 10 gives you 9.
4. If one node (computer/network connection/hard drive) goes down, the whole system keeps on going. Again, since the same data is stored on multiple drives, if one system goes down, the controlling machine or machines will see this and know to pull data from the backup source.
Looking at what I had listed earlier, if we take down drive 1, the data on drive 1 is also on drive 3. So drive 3 takes over serving drive 3 and 1 data and drive 2 keeps on going since it’s the original.
My main problem I have right now is finding another system that has a decent sized drive in it. I found a spare machine here with a nice 13GB drive in it, I know, it’s huge. But for real world tests, this is too small. I want at least an 80GB drive and while I had 2 not so long ago, I gave them both to my father in law because I saw no need to keep such small drives. I could use them now.
More to come when I have it. Stay tuned…
hihi, you make the same mistake I made about a year ago.
It is not correct that it’s a raid5-like setup an have 66% usable space.
If you are like me, you will say I am wrong, and try to prove it. I had long discussions on IRC to try to prove this. Anyways, it really took me some time to figure out I was wrong.
In short: afr of 2 volumes writes every file twice, so it uses double of its space, so it halves the capacity of the total volume
You can see this thread from the mailing list:
http://lists.gnu.org/archive/html/gluster-devel/2007-11/msg00128.html
The really big discussion must live in an IRC log somewhere
Thank you for the comment!
Well that’s one of the main reasons why I want to do some test runs to see exactly how it works. I only started reading up about this on Friday afternoon and prolonged setup until today. I’ve been busy all morning and haven’t had a chance to check it out any more…that and I need to find a spare drive somewhere. All in all, you are probably right.
Looking at that list doesn’t tell me much. The setup the guy had was similar to the striping I have now, not a redundant FS, just striped across all drives. I guess the only option would be to have something similar to Raid1. Not something I really want since I want to keep as much data as I can.
I guess I’m getting confused on what this example actually implies.
http://www.gluster.org/docs/index.php/Unify_over_AFR
Again that’s one reason why I want to setup a simple server for testing. Get to see it in action.
Wel yes, the original poster had a different setup, but that gave me the idea of the “networked raid5″-like system. I thought I found the holy grale, and it took me days to see I was wrong.
So with glusterfs you need the extra storage for every extra copy you want to make, but I still think glusterfs is a great system. A raid6 like translator is on the roadmap btw.
If you don’t need the performance, and want a more efficient usage of the available storage space in a redundant way, have a look at cleversafe.org. I never tested it, and it seems a lot more complex to setup, but it looks very interesting.
Agreed, interesting and complex.
I had read about a possible Raid6 setup down the road. Very excited about that.
I like the simplicity of GlusterFS from what I’ve read so far. But again, same as I posted earlier, the documentation listed on their site is a little vague. I’d like to dive into it as much as I can so I will be able to post a howto of my own.