Adding real-time data visualization to your ColdFusion apps

September 8, 2010 | By

One of my favorite things to do with Adobe Flex is build applications that  show real-time data. In this blog post, I will share what I’ve learned in the past few months about using ColdFusion as a source of real-time data and Flex combined with Google Maps API for Flash as a means of visualizing the data. The live map below is an example of what you can build using this technique. (If you can’t see the live map, you need to make sure you have Flash enabled). The map you see is a real-time view of activity on several blogs including my blog as well as blogs by James Ward, Christophe Coenraets, Michael Chaize, Anne Petteroe, Holly Schinsky, Ted Patrick, Kevin HoytRyan Stewart, Ray Camden and also hits to Flex.org. Every time one of these sites is viewed by someone, a dot appears on the map at their approximate location.  Give it a shot!   I’ll let you figure out which color marker goes with each blog.  Go hit one of the blogs or Flex.org and see yourself on the map (assuming your IP address is mappable). There is also a “LOAD LAST HOUR” button if you want to replay the last hour in accelerated time. Below is everything you need to add this type of visualization to any website or application.

UserGlobe Bloggers
– Watch more Videos at Vodpod.

Technologies used in this application:

  • ColdFusion 9.01 standalone running on port 8500 — fresh out-of-the-box install on Windows 7 (will also work with ColdFusion 8 ) — this blog post assumes that ColdFusion is installed at C:\ColdFusion9.
  • BlazeDS 4 (included with ColdFusion 9.01 – also works with BlazeDS 3 and LiveCycle Data Services)
  • Flex SDK 4.1 (all of this will work with Flex 3.x or Flex 4.0 too with minor changes)
  • Flash Builder 4
  • ColdFusion Builder

Introduction to publish/subscribe messaging

The easiest way to tap into real-time data is to employ a messaging paradigm called publish/subscribe messaging (commonly called “pub/sub”).  The website/blog activity is published to a message destination as small messages.  Below we will write some ColdFusion code that publishes data and then we will write some Flex code that subscribes to these published messages and does fun things with them.

Publishing messages with ColdFusion

ColdFusion uses the sendGatewayMessage() function to publish messages.  Before we can use this nifty function, we need to first configure a data service messaging gateway instance.

  1. Go to ColdFusion administrator and go to the Gateway Instances option under EVENT GATEWAYS.
  2. Create a new gateway instance using the gateway type “DataServicesMessaging“.
    1. For this example, I used “cfgw” as the name of our new gateway ID.
    2. For now, just point the CFC PATH to any CFC. By creating an “onMessage()” function, ColdFusion can receive messages through the gateway but for this article, we’re only going to focus on sending messages so it’s not necessary to have this function.
    3. The configuration file already exist in your default installation at C:\ColdFusion9\gateway\config\flex-messaging-gateway.cfg.
    4. IMPORTANT: You’ll need to edit the following config file (C:\ColdFusion9\gateway\config\flex-messaging-gateway.cfg) and comment out the line: host=”localhost” (see the comments in the flex-messaging-gateway.cfg for an explanation).
    5. Click “Add Gateway Instance” to save your work.
  3. Restart ColdFusion – check the cfserver.log for any gateway-related errors.

Now we can use the sendGatewayMessage() function to broadcast data.  The code below publishes a simple struct that contains “Hello World” in the body.

<cfset msg= structNew()/>
<cfset msg['destination']	= "ColdFusionGateway"/>
<cfset msg['body']       	= "Hello World"/>
<cfset sendGatewayMessage("cfgw",msg)/>

Notice the sendGatewayMessage() function uses our recently created “cfgw“.  Also, notice the ‘destination‘.  In this example, we are using the preconfigured destination called “ColdFusionGateway“.  If you want to see the definition of this destination, check out the following two files:

  • c:\ColdFusion9\wwwroot\WEB-INF\flex\messaging-config.xml — defines messaging destinations – you’ll see “ColdFusionGateway” already defined.  Near the bottom of the definition is a section for configuring the communication channel:

    The “cf-polling-amf” channel is defined in the services-config.xml file.
  • c:\ColdFusion9\wwwroot\WEB-INF\flex\services-config.xml — defines the individual services.

We’ll talk more about these configuration files later.

Now we have some simple ColdFusion code that publishes messages.  Next, we need to build something to consume them!

Subscribing to messages using a Flex Consumer

Below is a very basic Flex application that will subscribe to the same destination that we used above, “ColdFusionGateway”.   To try this out, simply create a new Flex project and copy the code below into the main mxml file.  Don’t worry about selecting an application server in this sample because we’re defining everything in the code.  For me, it’s just easier to understand when everything is in front of me rather than wired into a properties file somewhere.

<?xml version="1.0" encoding="utf-8"?>
<s:Application xmlns:fx="http://ns.adobe.com/mxml/2009"
			   xmlns:s="library://ns.adobe.com/flex/spark"
			   xmlns:mx="library://ns.adobe.com/flex/mx"
			   applicationComplete="init()">
	<fx:Script>
		<![CDATA[
			import mx.controls.Alert;
			import mx.messaging.events.ChannelEvent;
			import mx.messaging.messages.IMessage;

			private function init():void
			{
				consumer.subscribe();
			}

			private function onMessage(message:IMessage):void
			{
				trace("Message received:" + message.body);
				log.text += message.body + "\n";
			}

			private function onChannelConnect(e:ChannelEvent):void
			{
				trace("Connected!");
				log.text += "Channel connected!\n";
			}

			private function onChannelDisconnect(e:ChannelEvent):void
			{
				trace("Disconnected!");
				log.text += "Channel disconnected!\n";
			}
		]]>
	</fx:Script>
	<fx:Declarations>
		<mx:ChannelSet id="channelSet">
			<!-- Default Polling Channel -->
			<mx:AMFChannel url="http://192.168.1.60:8500/flex2gateway/cfamfpolling"/>
		</mx:ChannelSet>

		<mx:Consumer id="consumer"
			 channelSet="{channelSet}"
			 destination="ColdFusionGateway"
			 resubscribeAttempts="-1"
			 resubscribeInterval="2000"
			 message="onMessage(event.message)"
			 channelConnect="onChannelConnect(event)"
			 channelDisconnect="onChannelDisconnect(event)"
			 fault="Alert.show(event.faultString)"/>

	</fx:Declarations>
	<s:TextArea id="log" width="100%" height="100%"/>

</s:Application>

The only thing you’ll need to change is the IP address in the ChannelSet to point to your ColdFusion server.  I have event handlers that fire when the app successfully connects to the server, when a message is received and when the app gets disconnected from the server.  When you first run the app, after a few seconds, you should see the “Channel Connected!” message.  Now you can run the CF code above to send the “Hello world” message and you should see it received into your Flex application.  It might take a few seconds due to the type of channel we are using.  We will improve the response time later by creating our own custom channel definition.

Adding coolness step 1: Getting the IP address and converting to location

Now that you know how to publish data from ColdFusion and consume that data from a Flex application, it’s easy to build some really cool stuff.  First, we need to add some additional data to our outbound message from ColdFusion.

Getting the IP address in ColdFusion is super easy.  It’s already available to you in #CGI.REMOTE_ADDR#.  There are several services that provide IP address to location conversion but my favorite way to do it is to use a local database from MaxMind, a provider of geolocation and online fraud detection tools.  The conversion is lightning fast because the database is right on my server.  They have several databases available, but I highly recommend using one of the following:

MaxMind provides APIs for C, C#, Perl, PHP, Java, Python, Ruby, VB.NET, Pascal (huh?), and JavaScript.   Since Java is in the list, it’s easy to get this working with ColdFusion as well.  Thankfully, I didn’t have to do any work on this because Brandon Purcell has done it for us!  Brandon provides step-by-step instructions in his blog post.  To summarize Brandon’s steps, you download a JAR file that he created, copy it to your C:\ColdFusion9\runtime\servers\lib folder (CF restart required), add some code to your Application.cfc (example provided in Brandon’s post), and put the included geo.cfc file in your application folder.

Now we can do this:

<cfinvoke component="geo" method="ipLookup" returnVariable="location">
	<cfinvokeargument name="IP" value="#CGI.REMOTE_ADDR#"/>
</cfinvoke>

<cfset geodata = structNew()/>
<cfset geodata['destination']		= "ColdFusionGateway"/>
<cfset geodata['body']['city'] 		= "#location.IPCITY#"/>
<cfset geodata['body']['country'] 	= "#location.COUNTRYLONG#"/>
<cfset geodata['body']['countryshort']	= "#location.COUNTRYSHORT#"/>
<cfset geodata['body']['latitude'] 	= "#location.IPLATITUDE#"/>
<cfset geodata['body']['longitude'] 	= "#location.IPLONGITUDE#"/>

<cfset sendGatewayMessage("cfgw",geodata)/>
<cfdump var="#geodata#"/>

Notice that I added a to the end so I can see the results.  It’s very likely that if you run the code above, it will not show a location because your IP address is probably a private one since you are in the same subnet as your server.  To test, I put in my public IP address in place of #CGI.REMOTE_ADDR# and got the following:

Now we’re getting somewhere!  The message we are broadcasting now contains city, country (two formats), latitude, and longitude.  We can extract these new fields on the Flex side by referencing message.body.city, message.body.country, message.body.countryshort, message.body.latitude, and message.body.longitude.

Adding coolness step 2: Mapping the location

Now that we have lat/long, we can use many different mapping services.  You’ll find examples of Google, ESRI, MapQuest and Yahoo! Maps in Tour de Flex under the mapping category.  For this blog post, I’m using the Google 3D Maps API for Flash.  You’ll find many other Google Maps-related APIs at http://code.google.com/apis/maps.

I decided to use the 3D Map APIs for this because it allows you to change the map attitude, pitch, and yaw, which is a lot of fun and always impresses.

Steps to get this working:

  1. Go to http://code.google.com/apis/maps/documentation/flash and sign up for a Google Maps API Key (takes less than a minute and it’s free).
  2. Create a new Flex project like the one above (no special options, app server, etc. — just a plain ol’ Flex project).
  3. Download the Google Maps API for Flash SDK from the same page.  The ZIP file contains documentation and the map_flex_1_18b.swc that you need to place in your Flex project’s libs folder. (As of this writing the version was 1.18b; your SWC filename may vary.)
  4. Use the code below as your main mxml file.
  5. Don’t forget to modify the IP address in the to point to your ColdFusion server.
<?xml version="1.0" encoding="utf-8"?>
<s:Application xmlns:fx="http://ns.adobe.com/mxml/2009"
			   xmlns:s="library://ns.adobe.com/flex/spark"
			   xmlns:mx="library://ns.adobe.com/flex/mx"
			   xmlns:maps="com.google.maps.*"
			   width="100%" height="100%">

	<fx:Script>
		<![CDATA[
			import com.google.maps.InfoWindowOptions;
			import com.google.maps.LatLng;
			import com.google.maps.Map3D;
			import com.google.maps.MapEvent;
			import com.google.maps.MapOptions;
			import com.google.maps.MapType;
			import com.google.maps.View;
			import com.google.maps.controls.MapTypeControl;
			import com.google.maps.controls.NavigationControl;
			import com.google.maps.geom.Attitude;
			import com.google.maps.overlays.Marker;
			import com.google.maps.overlays.MarkerOptions;
			import com.google.maps.styles.FillStyle;

			import mx.controls.Alert;
			import mx.messaging.events.ChannelEvent;
			import mx.messaging.messages.IMessage;

			private function onMapPreinitialize(event:MapEvent):void
			{
				// Before the maps is rendered, we set a bunch of options
				var myMapOptions:MapOptions = new MapOptions();
				myMapOptions.zoom 		= 3.5;
				myMapOptions.center 	= new LatLng(10,-30);
				myMapOptions.mapType 	= MapType.SATELLITE_MAP_TYPE;
				myMapOptions.viewMode 	= View.VIEWMODE_PERSPECTIVE;
				myMapOptions.attitude 	= new Attitude(0,30,0);
				myMapOptions.backgroundFillStyle = new FillStyle({color: 0x000000, alpha: 1.0}),

				map.setInitOptions(myMapOptions);
			}

			private function onMapReady(event:MapEvent):void
			{
				map.addControl(new MapTypeControl());    // selection of map, satellite, hybrid and terrain
				map.addControl(new NavigationControl()); // Controls for pan, zoom, attitude

				consumer.subscribe();	// Start receiving messages now that the map is ready
			}

			private function onMessage(message:IMessage):void
			{
				var city:String			= message.body.city;
				var country:String 		= message.body.country;
				var countryshort:String = message.body.countryshort;
				var latitude:String		= message.body.latitude;
				var longitude:String	= message.body.longitude;

				// Sometimes the city and/or country come back blank.  I prefer "Unknown"
				if(city == "") city = "Unknown";
				if(country == "") country = "Unknown";

				var location:String		= city + ", " + country;

				// We need a LatLng object for use with the map
				var latlng:LatLng = new LatLng(message.body.latitude, message.body.longitude);

				// If the latitude/longitude is populated, go place the marker and an info window
				if(latlng != null)
				{
					// Customize our marker a bit (tons of options are available including a custom icon
					var markerOpts:MarkerOptions = new MarkerOptions();
					markerOpts.tooltip = location;
					markerOpts.radius = 5;
					markerOpts.fillStyle = new FillStyle({color:0xff0000});

					// Create a new marker with our customizations
					var marker:Marker = new Marker(latlng,markerOpts);

					map.addOverlay(marker);  // Add our new marker to the map

					// Customize the popup info window
					var infoWindowOpts:InfoWindowOptions = new InfoWindowOptions();
					infoWindowOpts.titleHTML = location;
					infoWindowOpts.contentHTML = latitude.substr(0,6) + "," + longitude.substr(0,6);
					infoWindowOpts.fillStyle = new FillStyle({alpha:0.5});

					map.openInfoWindow(latlng,infoWindowOpts); // Add the info window
				}
				trace("Message received from " + location);
			}

			private function onChannelConnect(e:ChannelEvent):void
			{
				trace("Connected!");
			}

			private function onChannelDisconnect(e:ChannelEvent):void
			{
				trace("Disconnected!");
			}
		]]>
	</fx:Script>

	<fx:Declarations>
		<mx:ChannelSet id="channelSet">
			<!-- Default Polling Channel -->
			<mx:AMFChannel url="http://192.168.1.60:8500/flex2gateway/cfamfpolling"/>
		</mx:ChannelSet>

		<mx:Consumer id="consumer"
					 channelSet="{channelSet}"
					 destination="ColdFusionGateway"
					 resubscribeAttempts="-1"
					 resubscribeInterval="2000"
					 message="onMessage(event.message)"
					 channelConnect="onChannelConnect(event)"
					 channelDisconnect="onChannelDisconnect(event)"
					 fault="Alert.show(event.faultString)"/>
	</fx:Declarations>

	<maps:Map3D id="map"
				mapevent_mappreinitialize="onMapPreinitialize(event)"
				mapevent_mapready="onMapReady(event)"
				width="100%" height="100%"
				key="PUT-YOUR-GOOGLE-API-KEY-HERE"/>

</s:Application>

Run the new Flex app and every time your ColdFusion code broadcasts a new activity message, it will extract the location from the message and display it on the Google 3D map.

You’ll find tons of fun features including the ability to “fly” to a new location, customize markers and more at http://code.google.com/apis/maps/documentation/flash/reference.html.   There are tons of other demos available at http://code.google.com/apis/maps/documentation/flash/demogallery.html

Adding coolness step 3: Using the tag

When I first started playing with this stuff, I was eager to add tracking to my own blog so I could watch my blog activity on a world map, but, my blog is not ColdFusion based so I didn’t have the luxury of adding a few lines of ColdFusion code to broadcast the needed data.  There are several server-side options that could work but my blog is hosted at WordPress.com so my options were very limited.  I don’t even have access to the server!  Late one night I had a crazy idea — I would use the tag!  It’s the most common way external assets are included in HTML.

With a little help from Ray Camden, I figured out how to get a CFM file to behave like an image.  On my blog, I simply have an image as follows:  .   Web browsers think it’s a 1×1 transparent png.

Here’s the source to trackimg.cfm:

<cfsetting enablecfoutputonly="Yes">
<cfsetting showdebugoutput="No">

<cfinvoke component="geo" method="ipLookup" returnVariable="location">
	<cfinvokeargument name="IP" value="#CGI.REMOTE_ADDR#"/>
</cfinvoke>

<cfset geodata = structNew()>
<cfset geodata['destination']			= "ColdFusionGateway">
<cfset geodata['body']['city'] 			= "#location.IPCITY#">
<cfset geodata['body']['country'] 		= "#location.COUNTRYLONG#">
<cfset geodata['body']['countryshort']	= "#location.COUNTRYSHORT#">
<cfset geodata['body']['latitude'] 		= "#location.IPLATITUDE#">
<cfset geodata['body']['longitude'] 	= "#location.IPLONGITUDE#">

<cfset sendGatewayMessage("cfgw",geodata)>

<!--- Read a 1x1 transparent png from disk that I created in photoshop --->
<cffile action="readBinary"
	file="c:\ColdFusion9\wwwroot\images\dot.png"
	variable="trackImage">

<!--- Return the 1x1 bitmap in png format --->
<cfheader name="content-disposition" value="Inline;filename=dot.png">
<cfcontent type="image/png;" variable="#trackImage#">

Now I can add tracking to any website I desire by simply adding the image tag.  In my demo app at the top of this blog post, you’ll notice that different sites show up as different colors.  I simply do this by adding a “color” variable to my geodata struct.  I set the color based on the domain in the referrer (#CGI.HTTP_REFERRER#) using code like this:

<cfelseif findNoCase("gregsramblings.com",#CGI.HTTP_REFERER#)>
	<cfset color="EEEE00">
	<cfset domain="gregsramblings.com">
<cfelseif findNoCase("flex.org",#CGI.HTTP_REFERER#)>
	<cfset color="00FF00">
	<cfset domain="flex.org">
<cfelseif findNoCase("jamesward.com",#CGI.HTTP_REFERER#)>
	<cfset color="9933FF">
...
...
<cfset geodata['body']['color']  = "#color#">
<cfset geodata['body']['domain'] = "#domain#">
<cfset geodata['body']['url']    = "#CGI.HTTP_REFERER#">
...
...

On the Flex side, I use message.body.color to set the marker color.  I use message.body.domain and message.body.url in the info window. There are lots of possibilities.

Geting Technical — Using messaging subtopics

All of the Flex code in this article is set up to receive all messages sent from the ColdFusion server.  If you later want to use the same ColdFusionGateway for multiple applications, you’re going to need some way to filter messages.  This is where subtopics come in.  Subtopics provide a means of categorizing messages within a destination.  By default, subtopics are enabled as you can see near the top of C:\ColdFusion9\wwwroot\WEB-INF\flex\messaging-config.xml.   To publish messages on a specific subtopic, add the following to your struct: .  On the Flex side, we add subtopic=”BLOGS” to our .  Now our Flex app will only receive messages sent specifically on the BLOGS subtopic.  Subtopics can be defined as mainToken[.secondaryToken][.additionalToken] so you could use something like “BLOGS.GREG” for one site and “BLOGS.BEN” on another.  On the Flex side, you can subscribe to either or you can use wildcards to subscribe to multiple subtopics  – subtopic=”BLOGS.*”.   When you subscribe to a subtopic, the filtering happens at the server, so your app only receives messages that match.

In addition to subtopics, you can also use selectors.  For more information on selectors and subtopics, refer to the BlazeDS documentation.

Getting Technical — Channel Types

You may have noticed that there is sometimes a delay of a few seconds after ColdFusion sends the message until the Flex app receives it.  So far we have been using the out-of-the box configuration for ColdFusionGateway.  If you look in the C:\ColdFusion9\wwwroot\WEB-INF\messaging-config.xml, you’ll see that the assigned channel is cf-polling-amf, which is defined in C:\ColdFusion9\wwwroot\WEB-INF\flex\services-config.xml.  This is set up to do basic polling, which is not super responsive.  We can speed things up significantly by implementing “long polling”.  Here are the steps:

1. Add the following channel definition to your C:\Coldfusion9\wwwroot\WEB-INF\services-config.xml:

<channel-definition id="cf-longpolling-amf" class="mx.messaging.channels.AMFChannel">
	<endpoint url="http://{server.name}:80/{context.root}/flex2gateway/amflongpolling" class="flex.messaging.endpoints.AMFEndpoint"/>
	<properties>
		<polling-enabled>true</polling-enabled>
		<polling-interval-seconds>3</polling-interval-seconds>
		<wait-interval-millis>60000</wait-interval-millis>
		<client-wait-interval-millis>1</client-wait-interval-millis>
		<max-waiting-poll-requests>200</max-waiting-poll-requests>
	</properties>
</channel-definition>

2. Edit the C:\ColdFusion9\wwwroot\WEB-INF\messaging-config and assign this new channel to the ColdFusionGateway destination. The modified XML will look like this:

<channels>
	<channel ref="cf-longpolling-amf"/>
	<!--<channel ref="cf-polling-amf"/>-->
</channels>

3. Modify your Flex code and change the URL of your ChannelSet:

<mx:ChannelSet id="channelSet">
	<!-- Default Polling Channel -->
	<mx:AMFChannel url="http://192.168.1.60:8500/flex2gateway/amflongpolling"/>
</mx:ChannelSet>

4. Restart ColdFusion and everything should work as before, but you’ll notice that messages get through faster.

Polling and Long Polling are not the only two options.  BlazeDS also supports HTTP Streaming, which has even better performance but some firewalls don’t like a persistent connection.  You can find more information about endpoints in Holly Schinsky’s blog post and in the documentation.

Getting Technical — What about LiveCycle Data Services?

LiveCycle Data Services ES2 (LCDS) offers the same messaging features as BlazeDS but it also has some additional capabilities that you need to be aware of.

  • RTMP: In addition to polling, long polling, and HTTP Streaming, LCDS provides Real Time Messaging Protocol (RTMP).  RTMP streams data in real-time over the TCP-based RTMP protocol in the binary AMF format.  It’s super fast and efficient and great for situations where you need thousands of subscribers to your message topics.  You can read more about channels and endpoints in LCDS here.
  • NIO-based endpoints:  Unlike BlazeDS endpoints, NIO-based endpoints are outside the servlet container and run inside an NIO-based socket server. NIO-based endpoints can offer significant scalability gains. Because they are NIO-based, they are not limited to one thread per connection. Far fewer threads can efficiently handle high numbers of connections and IO operations.  NIO-based endpoints are documented here.
  • Reliable messaging:  In LCDS, you can set a destination to be “reliable”.  Basically, LCDS will confirm that each message is received by all active consumers.  With non-reliable messaging, a short network outage could result in lost messages because there is no “retry” mechanism.  For applications like our activity tracking map, it’s no big deal..we just miss a dot on our map!  However, if you use messaging in an application where every message is critical, this is a key feature.  You can read more about it here.
  • Message throttling: Throttling is a feature that allows you to limit the number of server-to-client and client-to-server messages that a Message Service  destination can process per second.  Throttling is useful when the server could overwhelm slow clients with a high number of messages that they cannot process or the server could be overwhelmed with messages from clients.  You can read more about throttling here.

The full LCDS 3.1 documentation can be found at http://help.adobe.com/en_US/LiveCycleDataServicesES/3.1/Developing

Note:  I’ve only addressed the LCDS-specific features for messaging.  There are many other facets to LCDS that should not be ignored.  For messaging, it’s fairly simple.  If you need the features I just listed or if you need highly scalable messaging, then you need LCDS, otherwise BlazeDS will work for you.

What’s next with the real-time map app?

I’ve recently added server-side logging and the ability to replay history on the map (load last hour, load last day, etc.).  I’ve also added icons and other features.  I’ll do a new blog post showing the updated code in the next few days.

If you build something cool using the information in this blog post, I would love to see it.  I am dying to see a very high-traffic website on a map like this.

Here’s another example of real-time data on a map that I like to show — http://www.chessjam.com/online_chess_stats.cfm — what you will see is real-time activity from ChessJam, an online chess site that I helped develop.  The server is ColdFusion so I simply added code similar to what I show above in the function that handles chess moves.  Every time a player makes a move, it shows on the map.   I used an image of a map from 1812 as an overlay to give it a little age.  The really amazing thing here is that a map of the world could be made in 1812!  How did they do that with no GPS, no satellite pictures, etc.?  Crazy!

Enjoy!

Filed in: Adobe ColdFusion, Adobe Flex, Adobe LiveCycle Data Services, BlazeDS, Data Visualization | Tags: , , , , , , , , , , , ,

About the Author (Author Profile)

Greg is a developer advocate for the Google Cloud Platform based in San Francisco, California

Comments (1)

  1. Merritt

    Hi Greg. Can you post the snippet you used to initialize Brandon’s component in the Application.cfc file? I tried it a few different ways and can’t seem to get the right values passed to the oonstructor. I’m using CF9.01 on OSX. Thanks in advance.