Going into production is likely one of the biggest steps your application will ever undertake, and it can be a daunting process. There are often many unexpected chal- lenges that appear from out of the blue, and there are also problems that plague all applications and that still need solving. Yes, deployment can be a troubling and stress- ful time.
To minimize the stress of taking your Lift applications into production, this section looks at both tools and techniques that can hopefully assist in your deployment and save you some angst. This includes monitoring and dealing with your application once it has successfully been deployed. As you may or may not know, Twitter is a large user of Scala, and they released a lot of their code under open source projects. One of these projects is called Ostrich, and it’s used for monitoring and collecting statistics on your application. You’ll see how you can implement Ostrich in your own applica- tion and leverage its statistical reporting interface.
Before that, though, we’ll explore some of the built-in Lift components and items of functionality that can be useful when nearing production. This includes built-in snippets and coding techniques that can save you CPU cycles.
15.4.1 Built-in assistance
Lift, unfortunately, can’t help you a great deal with things that were not dealt with effectively during the development cycle, but it can help ease the pain of prepar- ing for deployment and help with solving commonly occurring problems. In this section, you’ll see how Lift can help you prepare and provide infrastructure to alter
1 In order to actually achieve this, you’d also need to employ the services of dynamic DNS or Amazon’s Elastic Load Balancing. This is a regular network issue, though, so it’s somewhat beyond the scope of this book.
365 Deployment tools and techniques
behavior between environments, and provide assistance in solving common prob- lems faced by applications in the wild. The first component we’ll be looking at is environment run modes.
PROPERTIES AND RUN MODES
Traditionally, one of the things that can be tricky with deploying applications is the configuration difference between development, testing, staging, and production sys- tems. You often see people either hacking together custom scripts or following deploy- ment procedures by hand to ensure they change all the configuration values relevant to that particular environment.
Lift has a solution for this problem. Part of the lift-util package is an object called Props. This object has a few functions, and one of those is to determine Lift’s run mode. What’s a run mode you might ask? Well, Lift uses these different modes to pro- vide different implementations and functionality for different parts of the develop- ment cycle. For example, if you were using convention-based snippets that are resolved via reflection, and Lift was unable to determine the class you wanted, Lift would display, when you’re running in development mode, a rather helpful box on the page where the snippet should be located, and it would inform you that the snip- pet you wanted was unable to be found. In production mode, however, no such mes- sage would be displayed to the user.
You can access the run mode in your application code simply by calling
import net.liftweb.util.Props val m = Props.mode
In this example, m is assigned a value type of the Props.RunModes enumeration. There are a bevy of convenience methods defined on Props to access the run mode, so check out the documentation for more specific information.
The run mode itself must be set early in the application boot phase, so much so that it’s generally supplied as a system property via the –Drun.mode=production style parameter to the JVM of the container. If you were running your application in an environment where you didn’t have access to the system-level properties, you’d need to ensure that you have a filter in front of the LiftFilter to set the run mode explic- itly using System.setProperty. This is somewhat crude, though, so it’s typically pref- erable to set the run mode as a JVM startup argument where possible.
TEMPLATE CACHING
When running in production mode, Lift makes a whole set of optimizations and alter- ations to the way it operates. One of these is the caching of page templates.
During development, each template is loaded, parsed, and processed for each request. This allows you to make running changes to the application and see immedi- ate feedback, but obviously you don’t want to change your code in production, so this is needless overhead. To that end, Lift implements a template cache so that the HTML templates themselves are only loaded on the first request, and then each subsequent request pulls the template markup from a predefined cache that can hold up to 500 templates inside an in-memory LRU cache.
If you wish to override the default template-caching behavior, simply do the following:
LiftRules.templateCache = Full(InMemoryCache(100))
Any caching implementation you wish to use must extend net.liftweb.util.Template- Cache. The rest is up to you!
CLIENT CONTENT EXPIRY
When rolling out different versions of your web application, it can sometimes be a struggle ensuring that the client browser has the latest version of all your static con- tent, like CSS and JavaScript. This can often cause problems in that when you make visual changes, the user’s browser may be using a cached version from when they pre- viously browsed the site. To them, at least, the user experience may be damaged.
This problem can be neatly sidestepped by making use of the with-resource-id snippet that’s built into Lift and that you can use in your templates. Any time you want to reference static files, simply surround them as shown:
<lift:with-resource-id>
<script type="text/javascript src="/path/to/file.js"></script>
</lift:with-resource-id>
The result here is that file.js would have a GUID appended to the end of its URI. This GUID is calculated once per application boot, so every time you roll out a new ver- sion of your application, you can be sure that the end user always sees the latest version. In addition, if you are deploying your application into a clustered environ- ment, you’ll want to alter the logic that generates these identifiers so that it’s consis- tent across that version or deployment, perhaps using the build number or hash. The logic for the resource ID generation can be customized via the LiftRules.attach- ResourceId hook.
EXCEPTION HANDLERS
In production, you’ll likely want to give some kind of “oops, something went wrong”
message in the very unlikely event that your application throws an exception. Fortu- nately, Lift provides a hook for this—all exceptions that occur bubble up through LiftRules.exceptionHandler. This allows you to intercept the error and handle it specifically, logging particular parts of the request if it was a certain exception type, for example.
Implementing your own exception handler is simple:
LiftRules.exceptionHandler.prepend {
case (Props.RunModes.Production, _, exception) =>
RedirectResponse("/error") }
In this example, the code simply returns a RedirectResponse, but as you can return any LiftResponse subtype, you could flush a template to the browser, or pretty much do anything you like.
367 Deployment tools and techniques
15.4.2 Monitoring
Once your application is in production, it can often be tricky to keep track of specific operations or to get live metrics from the system about how well it’s performing or what operations are being used a lot. For example, perhaps you want to know how many people are logged in at any one time, or perhaps you’d like to take a particular measurement about how long requests are taking to process. These types of metrics can be really helpful as an application grows and there are more moving parts.
Twitter runs quite a number of large-scale Scala systems to handle the fire hose of 140 million+ tweets per day; monitoring has become an important function for them.
Helpfully, Twitter has released a lot of their in-house projects as open source efforts, so that others can also benefit from them.
Ostrich is one such project (https://github.com/twitter/ostrich). Ostrich provides additional reporting facilities over and above what is offered by normal Java Manage- ment Extensions (JMX) that allow you to conduct three types of operations, as detailed in table 15.2.
Now that you know what’s possible with Ostrich, let’s put this into practice and look at how to implement Ostrich in your application to start collecting stats. The first thing you need to do is add the twitter repository and dependency to your project defini- tion, as shown:
val ostrich = "com.twitter" % "ostrich" % "4.1.0" % "compile"
val twitterRepo = "twitter-repo" at "http://maven.twttr.com/"
Don’t forget to call reload and update if you’re already running the SBT shell so that SBT recompiles the project and downloads Ostrich.
Table 15.2 The different Ostrich metric types
Metric type Description
Counters A counter is something that never decreases its numeric value; it’s a forever-increasing amount. Things that a counter could be applied to would be, for example, births. Every time a baby is born, the value will increase; it’s not possible for someone to be unborn, so the value could never go down.
Gauges Imagine yourself checking the oil in a car. You’d use a dipstick to get an indication of the oil level at that precise moment. If you checked it again the following day, the reading would likely be different. This is a gauge: a one-off reading of a specific thing at a specific time.
Metrics These typically measure the time it takes for n operation to occur. In the case of a web application, you may want to measure the duration of a particular resource request whose speed you were concerned about.
Labels A key/value pair that’s normally used to indicate the state of a specific system. For example, webservice=offline.
The next thing you need to do is alter your application Boot class so that during the application startup, the Ostrich server is also started and registered. Likewise, when your application shuts down, you need to close the Ostrich server gracefully.
The next listing shows the alteration to Boot.
class Boot {
import com.twitter.ostrich._, admin.{RuntimeEnvironment},
admin.config.{StatsConfig,AdminServiceConfig,TimeSeriesCollectorConfig}
object OstrichWebAdmin extends AdminServiceConfig { httpPort = 9990 statsNodes = new StatsConfig { reporters = new TimeSeriesCollectorConfig :: Nil } :: Nil lazy val service = super.apply()(new RuntimeEnvironment(this)) }
def boot { ...
OstrichWebAdmin.service
LiftRules.unloadHooks.append(
() => OstrichWebAdmin.service.foreach(_.shutdown)) ...
} }
First you need to define an Ostrich configuration object that specifies the settings Ostrich needs to run B. There is a whole range of possible options here, but in this example, the configuration defines the port number on which Ostrich’s HTTP inter- face will run. This will allow you to get text, JSON, and graph representations of the collected data. Next, the Boot class calls the lazy value defined in the configuration object, which causes the Ostrich service to load C. This same service value is used again to shut down the service when the Lift application closes D, ensuring a graceful shutdown of Ostrich.
Now that you have Ostrich set up and running in your Lift application, you’ll prob- ably want to start collecting some statistical data. The most common type of metric you’re likely to use is a counter. Let’s assume you have a situation in which you want to record the number of times that an event takes place. This is a fit with the counter style of metric outlined in table 15.2. All you need to do is define an identifier for this event and do the following:
import com.twitter.ostrich.stats.Stats Stats.incr("nameOfThisCounter")
This becomes exceedingly useful when you want to monitor specific aspects of your system, because you can just load up the monitoring graphs to check out what’s going
Listing 15.5 Ostrich initialization code in the Boot class
Configure Ostrich
B
Start Ostrich
C
Close Ostrich on shutdown
D
369 Deployment tools and techniques
on and get the high-level overview of what those metrics look like. In fact, all of the Ostrich metrics can be accessed via the HTTP port you defined in the configuration object, and you can get that information as either a range of graphs or as raw data.
Figure 15.5 shows an example of the graph detailing request duration, which you can see for yourself at http://127.0.0.1:9990/graph/?g=metric:request_duration.
In addition to these nice graphs, Ostrich also exposes all its data in JSON for- mat, so if you have a cluster of machines, you could feasibly have a cluster monitor that consumes individual node data from each Ostrich instance, and then aggre- gates that with a tool such as Ganglia (http://ganglia.sourceforge.net/). Ostrich essentially gives you a standardized interface from which to collate analytical infor- mation from your applications.
The following subsections detail the specific types of Ostrich metrics and show how to implement them in your Lift applications, complete with working examples.
USING GAUGES
It’s often useful to gain a snapshot of part of your application, monitoring specific parts of your app to take a one-time specific reading of an aspect. For example, you might like to know at any given time how many active sessions an instance is dealing with.
Lift has an object called SessionMaster that deals with the session setup and tear- down, and it provides a hook called sessionWatchers. This hook allows you to supply your own actor object that will be notified every 10 seconds with an updated list of ses- sions. You can supply a listener actor here to pipe that information to Ostrich as a gauge. The following listing defines a basic listener.
import net.liftweb.actor.LiftActor
import net.liftweb.http.SessionWatcherInfo
Listing 15.6 Example SessionMonitor implementation Figure 15.5 Example of an Ostrich graph
object SessionMonitor extends LiftActor { private var sessionSize = 0
protected def messageHandler = {
case SessionWatcherInfo(sessions) =>
sessionSize = sessions.size }
def count = sessionSize }
Here we simply create a singleton object that implements LiftActor. This is a Lift- specific implementation of the actor paradigm and requires no start call to be sent, like the actors in the Scala standard library do; simply reference the object, and the actor will do the right thing.
Because this object is extending LiftActor, you need to implement the message- Handler method to define the function that should execute upon receiving the SessionWatcherInfo instance from the SessionMaster hook. The first parameter being extracted within the messageHandler (called sessions) is a map of live sessions.
In this example, you simply want to get the size of that map and save it to an internal variable called sessionSize, so you ask the session map for its size B so that whenever the gauge is asked for the number of sessions, there will always be an answer, even if it’s a few seconds out of date.
Now that you have this implementation for the listener, you need to wire it up to the SessionMaster so that it receives notifications from Lift about the number of ses- sions and also implements the Ostrich gauge to collect the results. You simply need to do the following in your Boot class:
SessionMaster.sessionWatchers =
SessionMonitor :: SessionMaster.sessionWatchers Stats.makeGauge("current_session_count"){
SessionMonitor.count.toDouble }
The first definition prepends the SessionMonitor actor to the list of actors that the SessionMaster will notify with the list of sessions, whereas the latter definition config- ures the gauge measurement. Whenever the gauge is requested from the monitoring application, Ostrich will ask SessionMonitor to return the count.
METRICS
The last type of metric Ostrich supports is what is known as a metric. Metrics are col- lected as aggregates, and they include the number of operations performed, sum, maximum, and minimum, all of which are useful for determining the standard devia- tion of a particular activity’s results.
In a web application, you may want to record the duration of a particular resource request. Like the session hook, there are also facilities to hook into request timing.
The following listing shows a simple request timer.
Update session count
B
371 Case studies
import com.twitter.ostrich._, stats.Stats, admin.Service object RequestTimer extends Service {
object startTime extends RequestVar(0L)
def beginServicing(session: LiftSession, req: Req){
startTime(Helpers.millis) }
def endServicing(session: LiftSession, req: Req, response: Box[LiftResponse]){
val delta = Helpers.millis - startTime.is
Stats.addMetric("request_duration", delta.toInt) }
override def start(){}
override def shutdown(){}
override def quiesce(){}
}
This listing defines a simple object that has two methods: beginServicing and end- Servicing. These methods will be wired into the LiftSession.onBeginServicing and LiftSession.onEndServicing hooks respectively. The implementation here is rather simple: you assign a timer value into a RequestVar when the request starts pro- cessing, and then, in the endServicing method, the value held in the startTime vari- able is subtracted from the new time, which yields the duration delta. This data is then piped to Ostrich via the Stats.addMetric method.
The only thing remaining is to once again wire this into your application Boot as shown:
LiftSession.onBeginServicing = RequestTimer.beginServicing _ ::
LiftSession.onBeginServicing
LiftSession.onEndServicing = RequestTimer.endServicing _ ::
LiftSession.onEndServicing
This is nearly identical to the SessionMaster example in that LiftSession hooks define a list of executable functions that are called at the appropriate lifecycle stages of a request.
This concludes the section on tooling. You’ve seen many different things that you can use to both ease your deployment and gain critical application information when you actually get into the wild. With this in mind, you’ll now learn about some of the real-life Lift users who run large-scale deployments in commercial environments.