Grabbing External Data with a Proxy (YQL)

In my last installment, I covered loading external JSON files into your code. That works great, if you have reasonable control over the data format.

If you’re dealing with a vendor or open data source, you may not have the control you want to make a JSONP call (or the data is in XML instead of JSON). There’s a way around this by using a proxy.

If you have access to building PHP or any other pre-processing language, you can build your own proxy. If not, you have to get sneaky.

There are few open proxies out there, including one from Google. However, I’ve found one that’s far more robust and easy-to-use: Yahoo’s YQL.

Using their console, you can write queries that allow you to extract data from an external source and load it into your own JavaScript. For example, here is a way to get JSON formatted data for Denver International Airport. Just click “Test” when the console loads. Take note of the “Callback” box where you can set a callback function for your JSONP.

You get something like this:

{
 "query": {
  "count": 1,
  "created": "2013-11-11T17:27:03Z",
  "lang": "en-US",
  "diagnostics": {
   "publiclyCallable": "true",
   "url": {
    "execution-start-time": "1",
    "execution-stop-time": "339",
    "execution-time": "338",
    "content": "http://w1.weather.gov/xml/current_obs/KDEN.xml"
   },
   "user-time": "339",
   "service-time": "338",
   "build-version": "0.2.1997"
  },
  "results": {
   "current_observation": {
    "version": "1.0",
    "xsd": "http://www.w3.org/2001/XMLSchema",
    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
    "noNamespaceSchemaLocation": "http://www.weather.gov/view/current_observation.xsd",
    "credit": "NOAA's National Weather Service",
    "credit_URL": "http://weather.gov/",
    "image": {
     "url": "http://weather.gov/images/xml_logo.gif",
     "title": "NOAA's National Weather Service",
     "link": "http://weather.gov"
    },
    "suggested_pickup": "15 minutes after the hour",
    "suggested_pickup_period": "60",
    "location": "Denver Intnl Arpt, CO",
    "station_id": "KDEN",
    "latitude": "39.87",
    "longitude": "-104.67",
    "observation_time": "Last Updated on Nov 11 2013, 9:53 am MST",
    "observation_time_rfc822": "Mon, 11 Nov 2013 09:53:00 -0700",
    "weather": "A Few Clouds",
    "temperature_string": "61.0 F (16.1 C)",
    "temp_f": "61.0",
    "temp_c": "16.1",
    "relative_humidity": "21",
    "wind_string": "Southwest at 6.9 MPH (6 KT)",
    "wind_dir": "Southwest",
    "wind_degrees": "230",
    "wind_mph": "6.9",
    "wind_kt": "6",
    "pressure_string": "1021.6 mb",
    "pressure_mb": "1021.6",
    "pressure_in": "30.28",
    "dewpoint_string": "21.0 F (-6.1 C)",
    "dewpoint_f": "21.0",
    "dewpoint_c": "-6.1",
    "windchill_string": "60 F (16 C)",
    "windchill_f": "60",
    "windchill_c": "16",
    "visibility_mi": "10.00",
    "icon_url_base": "http://forecast.weather.gov/images/wtf/small/",
    "two_day_history_url": "http://www.weather.gov/data/obhistory/KDEN.html",
    "icon_url_name": "few.png",
    "ob_url": "http://www.weather.gov/data/METAR/KDEN.1.txt",
    "disclaimer_url": "http://weather.gov/disclaimer.html",
    "copyright_url": "http://weather.gov/disclaimer.html",
    "privacy_policy_url": "http://weather.gov/notice.html"
   }
  }
 }
}

So, how do we load it?

In that console, fill everything out for your data call. For my example, I am going to continue with the weather from DIA. I will grab the REST query at the bottom and use it for my AJAX call thusly:

var weatherStn = "KDEN";
$.ajax({
   url:"http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20xml%20where%20url%3D'http%3A%2F%2Fw1.weather.gov%2Fxml%2Fcurrent_obs%2F"+weatherStn+".xml'&format=json&diagnostics=true&callback=",
   dataType : 'json',                
   success: function(data){
      var c = data.query.results.current_observation;
      document.write("The temperature at "+c.location+" is " + c.temperature_string);
   }                
});

YQL LogoBe sure to play around with the console. There are some sweet features like selecting JSON or HTML as your data source. You can also set the returned data to be XML instead of JSON (if you so desire).

Related Links:

2 comments

  1. Cool. I’ve been trying to figure out how to work this without a proxy (our admin is being stubborn about it).

    Many thaks. I would love to see more on YQL, if you can.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.