by Jonathan Weiss

Starting Point
one machine
ruby / rails
mysql

Worst Case Popularity

problems vs users
exponential

difficult path
migrate from one server to multiple servers… if not made from the very beginning

PROBLEMS

Backup
high availability
redundancy
very big data sets

File system
important files must be accessed from many servers
NFS/Samba not practical

Spontaneous Traffic
slashdoted!
digged!

Load Fluctuation
load variability across different time slots
when not being used, you’re still paying for those resources

SOLUTION: Amazon Web Services?

S3 – Simple Storage Service
redundant data store
EC2 – Elastic Computer Cloud
virtual server per hour

S3
as much as you like
max 5Gb per object
organized in buckets
web-services API for uploads

Buckets
unique over all S3
contains many key-valued-metadata tuple
cannot contain other buckets
key can contain “/”
access control

S3 with AWS::S3
ruby package / class for interacting and manipulating S3 objects

S3 object manipulation

upload
S3Object.store(…)

download
obj = S3Object.find (..)

EC2
based on XEN virtualization
on demand virtual servers controlled with web service API
1,7GHz
1,75 RAM
160 Gb local file storage (non-persistant!!!)

use your favorite linux distr. (linux kernel 2.6.16)
amazon machine images (AMI) are stored on s3
ACLs for hosts/ports access control

EC2 Tools

list available images
ec2-describe-images

start a new instance
ec2-run-instances ami-xxxxx -k $RSA_KEY

login with ssh
ssh root@domU-xx-xx-xx-xx-xx.usma2.compute.amazonaws.com

shutdown instance
ec2-terminate-instance i-xxxxxxx

EC2 with Rails

amazon-ec2 gem

?? How does this solve my problems ??

S3-Backup
s3sync.rb
brackup
jungle disk
s3 fuse
s3dav
duplicity
s3Browser
Firefox S3 Organizer Extension

s3sync.rb (the most used)
modeled from rsync
backup
s3sync.rb -v -r -ssl
restore
s3sync.rb -v -r -ssl

S3 Asset Host
client static requests are sent to S3
client dynamic requests go to your main host
S3 Authenticated User Data
user request something
webservice request to s3
hosts replies with a redirect
user request the redirect link

try-it:
attachment_fu rails plugin

On-demand computing with EC2
load variability along the days/weeks

time-based with cron
0 18 * * * start_additional_instances.sh
0 18 * * * stop_additional_instances.sh

Load based with monit
tail /etc/monitrc

check system example.com
if cp usage (user) > 70% for X cycles then
exec start_additional_instances.sh
else-if cp usage (user) < 70% fro X cycles then exec stop_additional_instances.sh end client Apache load app server ec2 app server ec2 DB balancer app server ec2 app server ec2 Slave DB app server ec2 app server ec2 S3 monitor EC2 for extra capacity apache load balancer app server ec2 app server vpn app server ec2 DB app server ec2 own data-center amazon data-center problems? latency... Load Balancer / Proxy client apache mod_proxy balancer server server server server port8000 port8000 port8000 port8000 mongrel mongrel mongrel mongrel swiftiply proxy backbend clients not much security client apache clusterport 80 swiftiply proxy backendport 8000 client client client client swiftiplied swiftiplied swiftiplied swiftiplied mongrel mongrel mongrel mongrel install gem install swiftiply -y start swiftiply -c swiftiply.yml clustr_address: 0.0.0.0 cluster_port: 80 daemonize: yes ... swiftiply Mongrel gem plugin that patches mongrel transforms mongrel in swiftiply client experimental