Setting up high availability storage with MogileFS
** 1. Environment**
I used 4 Xen virtual images running ubuntu 8.04.
Two will run a tracker and the database, the other two will be the storage nodes.
Lets say the IP addresses will be:
192.168.0.195 192.168.0.196 192.168.0.197 192.168.0.198
2. Initial Setup
Install iptables:
apt-get install iptables
then apply initial setup
iptables -A INPUT -i lo -j ACCEPT iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -A INPUT -p tcp --dport ssh -j ACCEPT iptables -A INPUT -j DROP iptables-save -c > /etc/iptables.rules
you can save the rule for beyond machine reboot by adding the two lines below to /etc/network/interfaces
pre-up iptables-restore < /etc/iptables.rules post-down iptables-save -c > /etc/iptables.rules
install mysql and wget and perldoc
apt-get install mysql-server apt-get install wget apt-get install perl-doc apt-get install libio-aio-perl apt-get install subversion apt-get install build-essential
3. Common steps for installing MogileFS
retrieve the code (ripped of the how to wiki):
cd /usr/local/src mkdir mogilefs-src cd mogilefs-src svn checkout http://code.sixapart.com/svn/mogilefs/trunk
install perl dependencies
cpan Danga::Socket cpan Gearman::Client cpan Gearman::Server cpan Gearman::Client::Async cpan Net::Netmask cpan Perlbal cpan IO::WrapTie
install the servers:
cd mogilefs-src/trunk/server perl Makefile.PL make make test make install
At the moment the test seems to need mysql to be installed and user root without password, so some tests are skipped
you will need to install MogileFS::Client
(the tests expect a tracker to run locally on port 7001)
cd mogilefs-src/trunk/api/perl/MogileFS-Client perl Makefile.PL make make test make install
and some admin tools:
cd mogilefs-src/trunk/utils perl Makefile.PL make make test make install
4. Tracker install
create the database:
mysql -uroot -p mysql> CREATE DATABASE mogilefs; mysql> GRANT ALL ON mogilefs.* TO 'mogile'@'%'; mysql> SET PASSWORD FOR 'mogile'@'%' = OLD_PASSWORD( 'sekrit' ); mysql> FLUSH PRIVILEGES; mysql> quit
Create the schema
./mogdbsetup --dbname=mogilefs --dbuser=mogile --dbpassword=sekrit
(admin privilege is required for the initial setup, so if you're admin user is not root with no password, you will need to specify --dbroopassword and --dbrootuser)
create /etc/mogilefs/mogilefsd.conf:
db_dsn DBI:mysql:mogilefs db_user mogile db_pass ****** conf_port 7001 listener_jobs 5
create a mogile user:
adduser mogile
and starts the tracker under that user:
su - mogile mogilefsd
open a port for the tracker
iptables -A INPUT -p tcp --dport 7001 -j ACCEPT
5. Storage node
On the storage server, create a configuration file at /etc/mogilefs/mogstored.conf with the following:
httplisten=0.0.0.0:7500 mgmtlisten=0.0.0.0:7501 docroot=/var/mogdata
open a port:
iptables -A INPUT -p tcp --dport 7500 -j ACCEPT
iptables -A INPUT -p tcp --dport 7501 -j ACCEPT
register a new storage node:
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 host add mogilestorage --ip=192.168.0.197 --port=7500 --status=alive
it should now appears in the list:
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 host list
Add a device to the storage:
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 device add mogilestorage 1
and create the directory:
mkdir -p /var/mogdata/dev1
make sure /var/mogdata/* is owned by mogile:mogile
chown -R mogile:mogile /var/mogdata/*
6. Starting the storage server
as root:
mogstored --daemon
7. Starting the tracker
su - mogile mogilefsd -c /etc/mogilefs/mogilefsd.conf --daemon exit
8. Testing
check that mogilefs components are online:
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 check
Quick sanity check of the storage daemon:
~/Projects/mogilefs $ telnet 192.168.0.197 7500 Trying 192.168.0.197... Connected to 192.168.0.197. Escape character is '^]'. PUT /dev1/test HTTP/1.0 Content-length: 4 \n test HTTP/1.0 200 OK Content-Type: text/html Content-Length: 18 Server: Perlbal Connection: close 200 - OK Connection closed by foreign host.
create a domain
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 domain add mydomain
and a class
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 class add mydomain images
Quick sanity check of the tracker:
root@bbc-01:~# mogtool --trackers=127.0.0.1:7001 --domain=mydomain --class=images inject osname osname
on the store node, check /var/mogdata/dev1/0/000/000 for a file named xxxxxxxxxx.fid .
If the file exists it's all good.
9. setting up the second pair
Replay instructions 1 to 8, then:
when you've got the second storage set up, you will need to register the second storage and its device to all trackers:
mkdir -p /var/mogdata/dev2 mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 host add mogilestorage2 --ip=192.168.0.198 --port=7500 --status=alive mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 device add mogilestorage2 2 mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.196:7001 host add mogilestorage2 --ip=192.168.0.198 --port=7500 --status=alive mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.196:7001 device add mogilestorage2 2
you will also need to register the first storage and its device to the second tracker:
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.196:7001 host add mogilestorage --ip=192.168.0.197 --port=7500 --status=alive mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.196:7001 device add mogilestorage 1
sanity check the installation:
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001,192.168.0.196:7001 check
Checking trackers...
192.168.0.195:7001 ... OK
192.168.0.196:7001 ... OK
Checking hosts...
[ 1] mogilestorage2 ... OK
[ 2] mogilestorage ... OK
Checking devices...
host device size(G) used(G) free(G) use% ob state I/O%