Setting up high availability storage with MogileFS

* 1. Environment*

I used 4 Xen virtual images running ubuntu 8.04.
Two will run a tracker and the database, the other two will be the storage nodes.

Lets say the IP addresses will be:

192.168.0.195
192.168.0.196
192.168.0.197
192.168.0.198

2. Initial Setup

Install iptables:

apt-get install iptables

then apply initial setup

iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp --dport ssh -j ACCEPT
iptables -A INPUT -j DROP
iptables-save -c > /etc/iptables.rules
you can save the rule for beyond machine reboot by adding the two lines below to /etc/network/interfaces
pre-up iptables-restore < /etc/iptables.rules
post-down iptables-save -c > /etc/iptables.rules

install mysql and wget and perldoc

apt-get install mysql-server
apt-get install wget
apt-get install perl-doc
apt-get install libio-aio-perl
apt-get install subversion
apt-get install build-essential

3. Common steps for installing MogileFS

retrieve the code (ripped of the how to wiki):

cd /usr/local/src
mkdir mogilefs-src
cd mogilefs-src
svn checkout http://code.sixapart.com/svn/mogilefs/trunk

install perl dependencies

cpan Danga::Socket
cpan Gearman::Client
cpan Gearman::Server
cpan Gearman::Client::Async
cpan Net::Netmask
cpan Perlbal
cpan IO::WrapTie

install the servers:

cd mogilefs-src/trunk/server
perl Makefile.PL
make
make test
make install

At the moment the test seems to need mysql to be installed and user root without password, so some tests are skipped

you will need to install MogileFS::Client
(the tests expect a tracker to run locally on port 7001)

cd mogilefs-src/trunk/api/perl/MogileFS-Client
perl Makefile.PL
make
make test
make install

and some admin tools:

cd mogilefs-src/trunk/utils
perl Makefile.PL
make
make test
make install

4. Tracker install

create the database:

mysql -uroot -p
mysql> CREATE DATABASE mogilefs;
mysql> GRANT ALL ON mogilefs.* TO 'mogile'@'%';
mysql> SET PASSWORD FOR 'mogile'@'%' = OLD_PASSWORD( 'sekrit' );
mysql> FLUSH PRIVILEGES;
mysql> quit

Create the schema

./mogdbsetup --dbname=mogilefs --dbuser=mogile --dbpassword=sekrit

(admin privilege is required for the initial setup, so if you're admin user is not root with no password, you will need to specify --dbroopassword and --dbrootuser)

create /etc/mogilefs/mogilefsd.conf:

db_dsn DBI:mysql:mogilefs
db_user mogile
db_pass ******
conf_port 7001
listener_jobs 5

create a mogile user:

adduser mogile

and starts the tracker under that user:

su - mogile
mogilefsd

open a port for the tracker

iptables -A INPUT -p tcp --dport 7001 -j ACCEPT

5. Storage node

On the storage server, create a configuration file at /etc/mogilefs/mogstored.conf with the following:

httplisten=0.0.0.0:7500
mgmtlisten=0.0.0.0:7501
docroot=/var/mogdata

open a port:

iptables -A INPUT -p tcp --dport 7500 -j ACCEPT

iptables -A INPUT -p tcp --dport 7501 -j ACCEPT

register a new storage node:

mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 host add mogilestorage --ip=192.168.0.197 --port=7500 --status=alive

it should now appears in the list:

mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 host list

Add a device to the storage:

mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 device add mogilestorage 1

and create the directory:

mkdir -p /var/mogdata/dev1

make sure /var/mogdata/* is owned by mogile:mogile

chown -R mogile:mogile /var/mogdata/*

6. Starting the storage server

as root:
mogstored --daemon

7. Starting the tracker

su - mogile
mogilefsd -c /etc/mogilefs/mogilefsd.conf --daemon
exit

8. Testing

check that mogilefs components are online:

mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 check

Quick sanity check of the storage daemon:

~/Projects/mogilefs $ telnet 192.168.0.197 7500 Trying 192.168.0.197...
Connected to 192.168.0.197.
Escape character is '^]'.
PUT /dev1/test HTTP/1.0
Content-length: 4
\n test
HTTP/1.0 200 OK
Content-Type: text/html
Content-Length: 18
Server: Perlbal
Connection: close
200 - OK
Connection closed by foreign host.

create a domain

mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 domain add mydomain

and a class

mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 class add mydomain images

Quick sanity check of the tracker:

[email protected]:~# mogtool --trackers=127.0.0.1:7001 --domain=mydomain --class=images inject osname osname

on the store node, check /var/mogdata/dev1/0/000/000 for a file named xxxxxxxxxx.fid .
If the file exists it's all good.

9. setting up the second pair

Replay instructions 1 to 8, then:

when you've got the second storage set up, you will need to register the second storage and its device to all trackers:

mkdir -p /var/mogdata/dev2
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 host add mogilestorage2 --ip=192.168.0.198 --port=7500 --status=alive
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001 device add mogilestorage2 2
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.196:7001 host add mogilestorage2 --ip=192.168.0.198 --port=7500 --status=alive
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.196:7001 device add mogilestorage2 2

you will also need to register the first storage and its device to the second tracker:

mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.196:7001 host add mogilestorage --ip=192.168.0.197 --port=7500 --status=alive
mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.196:7001 device add mogilestorage 1

sanity check the installation:

mogadm --lib=/usr/local/share/perl/5.8.8 --trackers=192.168.0.195:7001,192.168.0.196:7001 check

Checking trackers...
192.168.0.195:7001 ... OK
192.168.0.196:7001 ... OK

Checking hosts...
[ 1] mogilestorage2 ... OK [ 2] mogilestorage ... OK

Checking devices...
host device size(G) used(G) free(G) use% ob state I/O%

Rija Ménagé

Read more posts by this author.

Subscribe to Pomme::TAB

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!
comments powered by Disqus