Servers A, B, and C each running a different rails app using a MySQL DB installed locally on each server. Had server D that should work as a slave for each of the DBs in order to have an up-to-date copy of the DBs in case of a HD crash. The backups of the DBs are also performed from the slave in order to avoid the locking of the DB on the prod servers in connection with the mysqldump command.

So how do we do that?

First step is to be able to run multiple MySQL instances on server D.

Seems that the preferred way to do this with MySQL 5.0 is to use the MySQL Instance Manager.
Unfortunately, the /etc/init.d/mysql script you get when installing MySQL on Ubuntu using
apt-get does not use the MySQL instance manager.

So I installed from source:
wget http://dev.mysql.com/get/Downloads/MySQL-5.0/mysql-5.0.85.tar.gz/from/http://mirrors.dotsrc.org/mysql/
tar xvzf mysql-5.0.85.tar.gz
cd mysql-5.0.85/
CFLAGS="-O3" CXX=gcc CXXFLAGS="-O3 -felide-constructors \
-fno-exceptions -fno-rtti" ./configure \
--prefix=/usr/local/mysql --enable-assembler \
--with-mysqld-ldflags=-all-static
make
sudo make install

Setup some symlinks
sudo ln -s /usr/local/mysql/bin/mysql /usr/local/bin
sudo ln -s /usr/local/mysql/bin/mysqldump /usr/local/bin
sudo ln -s /usr/local/mysql/libexec/mysqlmanager /usr/local/sbin

Installed the /etc/init.d/mysql script and made it use the MySQL Instance Manager
sudo sh -c "sed 's/use_mysqld_safe=1/use_mysqld_safe=0/' support-files/mysql.server > /etc/init.d/mysql"
sudo chmod 755 /etc/init.d/mysql
sudo update-rc.d mysql defaults

Installed the MySQL configuration file. Note that this lives in /etc/my.cnf and _not_ in /etc/mysql/my.cnf
sudo cp support-files/my-large.cnf /etc/my.cnf

Added the following to the top of /etc/my.cnf
[manager]
socket = /var/lib/mysql/manager.sock
pid-file = /var/run/mysql/manager.pid
password-file = /etc/mysqlmanager.passwd
monitoring-interval = 3600
user = mysql
log = /var/log/mysql/mysql-man.log
run-as-service

[mysql.server]
use-manager

Create the mysql user and the necessary directories
sudo groupadd mysql
sudo useradd -g mysql mysql
sudo mkdir -p /var/lib/mysql /var/run/mysql /var/log/mysql
sudo chown mysql:mysql /var/lib/mysql /var/run/mysql /var/log/mysql

Create the mysqlmanager password
sudo sh -c "mysqlmanager --passwd > /etc/mysqlmanager.passwd"
sudo chown mysql:mysql /etc/mysqlmanager.passwd
sudo chmod 600 /etc/mysqlmanager.passwd

Create the data directories
sudo /usr/local/mysql/bin/mysql_install_db --user=mysql --datadir=/usr/local/mysql/var/data
sudo /usr/local/mysql/bin/mysql_install_db --user=mysql --datadir=/usr/local/mysql/var/data1
sudo /usr/local/mysql/bin/mysql_install_db --user=mysql --datadir=/usr/local/mysql/var/data2

Replace the [mysqld] section in /etc/my.cnf with the following:
[mysqld]
datadir=/usr/local/mysql/var/data
port = 3306
socket = /tmp/mysql.sock
log-bin=mysql-bin
server-id = 10
relay_log = mysql-relay-bin
log_slave_updates = 1

[mysqld1]
datadir=/usr/local/mysql/var/data1
port = 3307
socket = /tmp/mysql1.sock
log-bin=mysql-bin
server-id = 11
relay_log = mysql-relay-bin
log_slave_updates = 1

[mysqld2]
datadir=/usr/local/mysql/var/data2
port = 3308
socket = /tmp/mysql2.sock
log-bin=mysql-bin
server-id = 12
relay_log = mysql-relay-bin
log_slave_updates = 1

Start the MySQL server
sudo /etc/init.d/mysql start

Connect to the MySQL Instance Manager
mysql -u root --socket=/var/lib/mysql/manager.sock -p

mysql> show instances;
+---------------+--------+
| instance_name | status |
+---------------+--------+
| mysqld | online |
| mysqld2 | online |
| mysqld1 | online |
+---------------+--------+


Yay!

Exit and connect to the MySQL DB running on port 3308:
mysql -u root -P 3308 -h 127.0.0.1
Note that you need to specify the host (-h) option. Otherwise, the mysql command will ignore the port option and just connect to the default instance running on port 3306.

I will make a followup post on how to setup the actual replication. Hope someone finds this useful

The Rails project I am currently working on is developed using BDD. This means that is has a big, fat spec suite. Or to be more specific it has 10033 examples!

This is very nice except for one thing: It is slooow

On my shiny (literally) new 2.4 GHz MacBook Pro the spec suite has a runtime of 670 seconds ie. 11 minutes and 10 seconds – yikes! This is with the ruby interpreter shipped with Mac OS X Leopard.

I have noticed running top that it seems to be mostly CPU bound. The ruby process hovers at around 95-100% CPU usage.

Ruby Enterprise Edition to the rescue!

Previously, I have tried to run the spec suite with the 1.8.6-20080810 version of REE and it did not change the runtime significantly.

The new 1.8.6-20081205 version has some interesting changes. First of all, the tcmalloc memory allocator now works with Mac OS X. And second of all, it has integration with the RailsBench garbage collector patches which allows for tweaking the GC settings of the ruby interpreter.

So what does mean in “real life”?

I downloaded and installed the new version of REE and ran the spec suite. The runtime with the new REE version was 436 seconds ie. 7 minutes and 16 seconds, chopping of nearly 4 minutes – VERY nice!

RailsBench GC patches to the rescue!

I decided to experiment a little with the GC settings ie.

export RUBY_GC_MALLOC_LIMIT=64000000

and reran the spec suite. The result: 221 seconds ie. 3 minutes and 41 seconds. Tried RUBY_GC_MALLOC_LIMIT=256000000 and the result: 209 seconds ie. 3 minutes and 29 seconds – holy Batman!

Thank you guys!

I suggest you go to workingwithrails.com and recommend Hongli Lai, Ninh Bui and Stefan Kaes like I just did – they deserve it.

The presentation was about adding full text search to a Rails app.

Here is a pdf with my presentation:

Adding Full Text Search to Your Rails App

The conference was arranged by the same crew doing the JAOO conference and most (all?) presentations were recorded on video. When the videos are available online I will post a link.

Nice! Let’s play with it! Please notice that I am located in Denmark and that all tests were performed on my 2048/512 ADSL line.

Download the new version of Amazon S3 Authentication Tool for Curl

Unzip it and create an .s3curl file containing you AWS keys as described in the readme file.

Now let’s create some buckets – a US bucket and an EU bucket:

s3curl.pl --id personal --createBucket -- http://s3.amazonaws.com/erichsen.net.us
s3curl.pl --id personal --createBucket=EU -- http://s3.amazonaws.com/erichsen.net.eu


Fetch some test files a 50K file and a 10MB one:

wget ftp://ftptest1.tele.dk/pub/50Ktest.rnd
wget ftp://ftptest1.tele.dk/pub/10Mtestb.rnd


And upload them:

s3curl.pl --id=personal --acl public-read --put 10Mtestb.rnd -- http://erichsen.net.us.s3.amazonaws.com/10Mtestb.rnd
s3curl.pl --id=personal --acl public-read --put 50Ktest.rnd -- http://erichsen.net.us.s3.amazonaws.com/50Ktest.rnd
s3curl.pl --id=personal --acl public-read --put 10Mtestb.rnd -- http://erichsen.net.eu.s3.amazonaws.com/10Mtestb.rnd
s3curl.pl --id=personal --acl public-read --put 50Ktest.rnd -- http://erichsen.net.eu.s3.amazonaws.com/50Ktest.rnd


Try fetching the large file from the US bucket a couple of times

ab -n 1 http://erichsen.net.us.s3.amazonaws.com/10Mtestb.rnd
...
Time taken for tests: 50.325 seconds
...
Transfer rate: 208.37 [Kbytes/sec] received

ab -n 1 http://erichsen.net.us.s3.amazonaws.com/10Mtestb.rnd
…
Time taken for tests: 48.351 seconds
…
Transfer rate: 216.87 [Kbytes/sec] received
And the EU bucket

ab -n 1 http://erichsen.net.eu.s3.amazonaws.com/10Mtestb.rnd
...
Time taken for tests: 47.907 seconds
...
Transfer rate: 218.88 [Kbytes/sec] received

ab -n 1 http://erichsen.net.eu.s3.amazonaws.com/10Mtestb.rnd
…
Time taken for tests: 50.943 seconds
…
Transfer rate: 205.84 [Kbytes/sec] received
With respect to transfer rate they seem to perform about the same from my local machine’s point of view. But I guess that this is what is to expect. The EU bucket should give better response times and for large files the response times are only a small fraction of the total transfer time.

But what about the small file?

US bucket

ab -n 50 http://erichsen.net.us.s3.amazonaws.com/50Ktest.rnd
...
Time taken for tests: 60.308 seconds
...
Time per request: 1206.16 [ms] (mean)
...


EU bucket

ab -n 50 http://erichsen.net.eu.s3.amazonaws.com/50Ktest.rnd
...
Time taken for tests: 26.676 seconds
...
Time per request: 533.52 [ms] (mean)


Now we’re talking!

Summary: For large files you could just as well use the US variant of S3. If you use S3 for serving the static files of your web site and most of your visitors come from Europe switching to the EU S3 should give your users significantly better load times.

This blog posting describes how I created an Ubuntu 6.06 LTS server edition image for use with Amazon EC2.

Download and install the EC2 command line tools

curl -O http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
mkdir ~/.ec2
cd ~/.ec2
unzip ec2-api-tools.zip
ln -s ec2-api-tools-1.2-13740 ec2-api-tools

Set the environment variables necessary to run the tools

export EC2_HOME=~/.ec2/ec2-api-tools
export PATH=$PATH:$EC2_HOME/bin
export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/

Download your private key and certificate from your Amazon Web Services account to the ~/.ec2 folder.

Generate a key pair

export EC2_PRIVATE_KEY=~/.ec2/pk-8WU9XGOPO65IKA7O96M2KEKVOS5288KU.pem
export EC2_CERT=~/.ec2/cert-8WU9XGOPO65IKA7O96M2KEKVOS5288KU.pem
mkdir ~/.ec2
ec2-add-keypair gsg-keypair > ~/.ec2/id_rsa-gsg-keypair
chmod 600 ~..ec2/id_rsa-gsg-keypair


Launch a Fedora Core 4: Base instance.

ec2-run-instances ami-20b65349 -k gsg-keypair

This returns an instance number like i-9536dcfc.

Try running

ec2-describe-instances i-9536dcfc

until the status returned is no longer ‘pending’ but ‘running’.

Allow ssh access and log in

ec2-authorize default -p 22
ssh -i ~/.ec2/id_rsa-gsg-keypair root@ec2-67-202-21-218.compute-1.amazonaws.com


On the EC2 instance run

wget http://erichsen.net/blog/fc4-base
chmod 755 fc4-base
./fc4-base

fc4-base is a script found in this forum posting. I adapted it to create an Ubuntu 6.06 image instead of 6.10.

After the script has finished execution copy the private key and certificate from the local machine to the EC2 instance

scp -i ~/.ec2/id_rsa-gsg-keypair ~/.ec2/pk-8WU9XGOPO65IKA7O96M2KEKVOS5288KU.pem root@ec2-67-202-21-218.compute-1.amazonaws.com:/root/
scp -i ~/.ec2/id_rsa-gsg-keypair ~/.ec2/cert-8WU9XGOPO65IKA7O96M2KEKVOS5288KU.pem root@ec2-67-202-21-218.compute-1.amazonaws.com:/root/


Create an image and sign it with the private key

ec2-bundle-image -i /mnt/ubuntu606base.img -k /root/pk-8WU9XGOPO65IKA7O96M2KEKVOS5288KU.pem -c cert-8WU9XGOPO65IKA7O96M2KEKVOS5288KU.pem -u '5171-9220-6573'


The image must be stored on S3 so I create a bucket (from my local Mac)

sudo gem i aws-s3 -y
export AMAZON_ACCESS_KEY_ID="1IQ8AHOAWNRDMQOI91ZK"
export AMAZON_SECRET_ACCESS_KEY="VCddmbA9C4D8w/mw6aLZjzCkMoEyx5EUvouJdY/4"
s3sh
Bucket.create('erichsen.net')

On the EC2 instance I upload the image

ec2-upload-bundle -b erichsen.net -m /tmp/ubuntu606base.img.manifest.xml -a '1IQ8AHOAWNRDMQOI91ZK' -s 'VCddmbA9C4D8w/mw6aLZjzCkMoEyx5EUvouJdY/4'


From my local Mac I register the instance and that’s it!

ec2-register erichsen.net/ubuntu606base.img.manifest.xml


The ec2-register returns an AMI id – in this case ami-4acd2823. Let’s try it out

ec2-run-instances ami-4acd2823 -k gsg-keypair
ec2-describe-instances i-9536dcfc
ssh -i ~/.ec2/id_rsa-gsg-keypair root@ec2-67-202-24-151.compute-1.amazonaws.com


YES! I was able to ssh into an EC2 instance running my Ubuntu 6.06 image! Note that the ssh configuration of the image is not the best – it allows root logins which is not in general a good idea.

Hope this helps someone else wanting to play with Ubuntu on EC2.

Boy, things have come a long way since I first installed SLS Linux on my pc using floppy discs back in 1994
These days I prefer Ubuntu 6.06 LTS server edition. It is supported until 2011 – nice that you don’t have to reinstall the server in a year or two. Furthermore, deprec makes it extremely simple to setup a Rails stack on an Ubuntu server.

Webserver

nginx. Nginx is fast, stable, lightweight and has easy configuration.

Load balancer

Usuyally I use the one built in to nginx.
Alternatives: Pen or HAProxy.

Rails-server

Well, Mongrel of course. In a cluster handled by Mongrel cluster.

Database

MySQL. It’s the database used by most Rails applications. Furthermore, it has some nice scaling support with master-slave setups. A couple of years ago I used and liked PostgreSQL a lot. I still prefer PostgreSQL’s query analyzer to MySQL’s.