Since I have been using Scalr to manage my Amazon Web Services farms I have been wanting more monitoring in terms of statistical information on services, traffic, disk usage, and uptime to name a few. Scalr has built in means of basic event notifications such as host up, host down, etc. Along with providing very basic load statistic via RRDtool. In the past I have always used Zabbix for most projects I have worked on so I wanted to be able to use it with Scalr. I am still testing the setup I am going to speak of so please keep that in mind. This is NOT a howto, but more of a brainstorming of how I plan on getting Zabbix integrated into my Scalr setup. In the Zabbix documentation (PDF) there are a few ways to use the auto-discovery that they cover (page 173). You can have Zabbix monitor a block of IPs to find new Zabbix Agents running for example. So here is what I will have my Zabbix Server do:
- Look for new Zabbix Agents on my AWS internal IP range.
- If the system.uname contains “Scalr” it will add to Scalr server group
- Server must be up for 30+ minutes
There will be other stipulations in order to get the server added to Zabbix. I will have system templates for each of my Scalr AMI roles. Once the server is added to Zabbix it will add them to to their respective groups and monitor for items and triggers listed in the system template. There will also be a rule to remove old instances after 24 hours from Zabbix after receiving the host down trigger. This way I will not have a bunch of old instances that were once monitored still cluttering Zabbix database. If you happen to also have Windows AWS instances you can add a rule to monitor these as well. The AMI just needs to have the Zabbix Windows Agent installed.
I have been playing around with the AWS Console recently released. It is a good start to a nice AWS provided interface for controlling EC2. It seems to only make sense that they provide a console instead of forcing people to look elsewhere such as RightScale or Scalr. For that matter I am not sure why Amazon does not just buy RightScale and provide their services as part of AWS.
I came across Scalr by accident when I was browsing projects in Google Code. It appears as though Scalr has become a pay service to manage your AWS instances along similar lines to RightScale. But the main difference is that Scalr charges a scant $50 a month. From the Scalr Google Code page:
Scalr is a fully redundant, self-curing and self-scaling hosting environment utilizing Amazon’s EC2.
It allows you to create server farms through a web-based interface using prebuilt AMI’s for load balancers (pound or nginx), app servers (apache, others), databases (mysql master-slave, others), and a generic AMI to build on top of.
The health of the farm is continuously monitored and maintained. When the Load Average on a type of node goes above a configurable threshold a new node is inserted into the farm to spread the load and the cluster is reconfigured. When a node crashes a new machine of that type is inserted into the farm to replace it.
Multiple AMI’s are provided for load balancers, mysql databases, application servers, and a generic base image to customize. Scalr allows you to further customize each image, bundle the image and use that for future nodes that are inserted into the farm. You can make changes to one machine and use that for a specific type of node. New machines of this type will be brought online to meet current levels and the old machines are terminated one by one.
I would love to hear some comments from those already using the service and how it compares to RightScale.
Saw this article when looking for information on using OpenVPN with Amazon Web Services. It is not exactly what I am looking to do with my latest project. But it is some good helpful information for those with existing infrastructure looking to use AWS. There is also VPN-Cubed which is another option if looking for a supported product. I have not used it but would love to hear some comments by those who have.
I am a big fan of using Subversion for things other than just versioning code. In the past I have used SVN to manage configuration files across many servers. Making it easy to deploy and (if needed) roll back changes. It is also a big help on development servers for PHP developers to commit changes and see them live on the development server. This is easily accomplished using SVN hooks. There are plenty of HowTos on this topic if seeking that information.
Now that I am up to my eyeballs in Amazon Web Services I am looking to use SVN to help me leverage the new found power of the cloud. Now I am not saying it is a good thing to use SVN for things other than code versioning. But it has always worked for me in many other ways as well.
Some of the things I am thinking of using SVN:
- Update DEV web server using hook scripts for devs to see changes to trunk.
- Maintain Apache and other config files for AMIs.
- Maintain code repository for versioning along with take advantage of S3 for backup and processing power of EC2.
This is a work in progress so I am looking to perfect the design to my liking soon.
I have been working on some new projects since leaving my last job about 6 months ago. One is to build an entire infrastructure that is highly available and redundant. With Amazon Web Services this is a snap and almost makes my job obsolete. With EC2, S3, EBS, and CloudFront you can build a scalable solution with dependable backups with ease. My goal now is to also use AWS to create an intranet for the company. I have not found much on this topic so I ma taking the time to document it here. My ideas so far:
- Use Fedora Directory Server as main LDAP solution. This can be used with EBS (Elastic Block Storage) with striped volumes to store the LDAP data. Along with having redundant multi-master replication geographically depending on where the EC2 instance is set.
- Subversion server using EBS for storage. Using EC2 for something along the lines of SVN is a good use since it speeds up performance. Along with having the backup ability of EBS and EBS Snapshots
- Twiki as documentation for intranet.
- SugarCRM for customer relationship management which will include project management and bug tracking.
- S3 for backing up users data automatically. This is helpful since it is a telecommuting company. Using a tool like JungleDisk or similar.
- The DEV and Staging environments will also be on AWS with same Amazon Machine Images (AMIs) as the production environment.
That is all I have so far I will update as the project comes along.