EC2 Stories

Here are some EC2 stories to drive SmartFrog component development

Define an AMI account

• We don't want to duplicate AMI security information around the files, so a password provider could be used to provide the keys needed to connect. For WS-* based connectivity, its pretty fiddly, but the HTTP QUERY api is simpler. All EC2 components should get their information from the same AMI login components.

Work with the S3 Filestore

This does not need the Ec2 filestore; we can work with S3 from behind the firewall

• Components to do better HTTP operations (PUT, POST, DELETE)
• Components to create buckets, push up data (progress? cancel?)
• Tests that can run during a Cruise Control build (billing?)

Update 2008-03-18: this has been implemented by way of the Restlet API; the underlying components can be used to work with any web site.

Persistence components that save tuples of data to S3 or other places.

Upload an AMI to S3

One or more (big) files is uploaded to S3 Prior to other deployments. This takes time and, given storage costs, is not something you want to do before every run.

Register an AMI image

After the upload, the file is turned into an AMI image

Create a firewall policy

Register a set of firewall policies for a host; it could just be a list of ports to open to everyone.

Deploy a public AMI

1. A public AMI is deployed
2. when the component is terminated, the AMI is terminated

Deploy a private AMI.

A private AMI is deployed

1. you configure a (local) sf file with the #of nodes and AMI login information
2. you run something on the command line (sfStart) to deploy local components to deploy a remote node
3. when the components are disconnected, the remote nodes stay live (production)
4. when the components are disconnected, the remote nodes shut down (dev)

Customise a deployed AMI

A custom or public AMI has been deployed. Customise it by installing new applications. This could be RPMs and apt-get, or SCP upload of Java componentry

Add customisation data to a deployment

When you deploy a component, you can put out context-specific information. This can be retrieved by custom GETs against a http://169...URL.

1. Provide a means to set this data on a deployment
2. Provide a means to query this data, turning it into attributes on the graph
3. For testing under CC, we could simulate the server.

Monitor an AMI

An AMI is running/has been started. Whenever this AMI is no longer pingable (by some custom definition of liveness) it should be terminated and restarted.

Deploy onto an AMI using SmartFrog (no long-haul RMI)

Rather than talk to SmartFrog using long-haul RMI, we want to deploy to it using the deploy-by-copy improvements SFR:SFOS-149
This would let us deploy things using SCP

Remote Management with JMX

• deploy the JMX web pages (with security!)
• manage SmartFrog

Remote management with RMI

putting security aside, can we remotely manage an EC2 system using Java RMI? From home? from HP?

Remote management with a REST API

Needs a RESTy API

Register a deployed node with DynDNS.org

We should be able to use our Liveness page component to do this.

• Dyndns.org provide A developer API to update URLs, you just do a GET with various parameters, and the back end updates their entry
• they provide test accounts for applications
• free accounts are easy to create
• Steve has antbook.org managed through them: With his password we can register lots of machines

Register a deployed node with a public XMPP server -jabber.org or talk.google.com

We have XMPP components that can do this; we just need to bind them to the public servers, and to get their login/passwords from somewhere.

Once registered, components can talk to each other over XMPP. We could have standard operations (GET, PUT. POST, DELETE) to exchange data with other components, and so share private information.

#2 Demo Application: private AMI running Smartfrog

• AMI with SmartFrog already running (private keys)
• Deploy AMI automatically starts SmartFrog (init.d)
• Bring up an application server+Database
• deploy an XMPP status component
• deploy a dyndns component to give a live DNS address