Apr 30, 2011

matrix.js introduction

Essentially, matrix.js is like other script loader, but it does other things such extensible resource type, dependencies registration and reload, resource locations, resource release and others. In the following, I will explain what the library can do. All these example presented here can be found at GitHub

The most used method in matrix is the "matrix" method. Here is the definition.

//returns a promise
//resourceKeys is a string or an array, e.g "a.css, b.css" 
//or ["a.css", "b.css"] or ["c.js", "a.css, b.css"]
matrix(resourceKeys[, loadByOrder])

If you have no configuration, the resource key is is treated as url of the resource. Resource key follow the format of [resourceName].[resourceType]. There are three build-in resource handler in matrix, "module", "js", and "css", resourceKeys can be a string or array. Most of the time, you should use string format resourceKeys. If it is a string, this means resource are fetched in parallel, if it is an array, this means resources are fetched in serials, most of time, we use resourceKeys as string. I will discuss more about it later. Just like any other script loader, you can load javascript and css like the following.

var promise = 
matrix("/css/reset.css, /css/base.css, /js/utilities.js, /js/app.js", 
).done(fn);

Behind the scene, the loader will fetch two css and two javascript files using ajax call in parallel, and evaluate each source file into the page when it is ready at client side. and run fn1, fn2, fn3 after all resources are loaded. It is important to know that, the order of making ajax call is not necessary the order when ajax call is finished, so the order of evaluation is not guaranteed. If the order of evaluation is important, you can use the following code.

var promise = 
matrix("/css/reset.css, /css/base.css, /js/utilities.js, /js/app.js", 
true).done(fn);

Behind scene, the load method actually convert it into the following call.

//reset depends nothing
matrix.depend("/css/reset.css", null); 

//base.css depends on reset.css
matrix.depend("/css/base.css", "/css/reset.css"); 

//utitlies.css depends on base.css
matrix.depend("/js/utilities.css", "/css/base.css");  

//app.css depends on utilities.css
matrix.depend("js/app.css", "/js/utilities.css");  
var promise = matrix("/js/app.js", true).done(fn);

Dependencies between the module is important in resource loading, sometimes is more complicated than serialized dependencies like above case. When loading resource, matrix follows a couple steps. Firstly, matrix determines the type of resource and the correct handler for processing the resource request. After that, matrix use the handler to execute three tasks(fetch, parse and evaluation). Fetch is to get the content ready at the client side. Parsing is an optional step, matrix currently extract dependencies from the content in this step, and this is necessary only when dependencies information is not pre-configured, and parsing method is implemented by the handler. Evaluation is to integrate the resource into the page. These three tasks are implemented by a resource handler.

Let's try a more complicated case. jQuery UI library offer lots of widgets. It is a highly modularized library, some components(like ui.mouse.js) are reused over the library like. If you only want to use a certain widget,e.g dialog, you can go to their home page, deselect all, and just select dialog, and it create a a package that include just the css, js file that dialog widget use. This is a great feature. But there is still a problem. If I have two page, page one need a,b,c widget, and page two need b,c,d widget, so I need to create two js package. To save the trouble, I might better off just create a package file that include all the modules. However this might load too many modules that I don't need. We can use matrix to load the minimum set of resource to be able to use a widget. First I will show you the hard way, later I will show you easy way. But regardless of which, we need to solve two problem, resource dependencies and and resoure locations.

To understand dependencies in jQuery UI, you need to download all the source of jQury UI. Open the source file in jQuery.ui.dialog.min.js, the header says

/*
 * jQuery UI Dialog 1.8.10
 *
 * Copyright 2011, AUTHORS.txt (http://jqueryui.com/about)
 * Dual licensed under the MIT or GPL Version 2 licenses.
 * http://jquery.org/license
 *
 * http://docs.jquery.com/UI/Dialog
 *
 * Depends:
 * jquery.ui.core.js
 * jquery.ui.widget.js
 *  jquery.ui.button.js
 * jquery.ui.draggable.js
 * jquery.ui.mouse.js
 * jquery.ui.position.js
 * jquery.ui.resizable.js
 */

Now we can see what dialog depends, dialog also depends jquery.ui.dialog.css. Let's change it for now and repeat this for other module accordion, autocomplete, button, datepicker, dialog, progressbar, resizable, selectable, slider, tabs, for example, add jquery.ui.accordion.css in jQuery.ui.accordion.min.js header. We will need it later in using next method.


/*
..
* Depends:
 * jquery.ui.dialog.css
 * jquery.ui.core.js
 * jquery.ui.widget.js
 *  jquery.ui.button.js
 * jquery.ui.draggable.js
 * jquery.ui.mouse.js
 * jquery.ui.position.js
 * jquery.ui.resizable.js

*/

To convert this into code, we can write the following. Please note that I am not using the url of resource as the resource key, later I will map the reource key to url.

matrix.depend( "ui.dialog.js": "ui.dialog.css, ui.core.js, ui.widget.js, ui.button.js, ui.draggable.js, 
ui.mouse.js,ui.position.js, ui.resizable.js");

But this is not enough, because inner resources have their own dependencies as well. So we need to open all direct and indirect referenced js files and find out their dependencies. If there a file has no dependencies, we still need to use a null value. For example, ui.core.js does not have any dependencies, so its dependencies is null. Finally, we can write something like the following.

matrix.depend( {
 "ui.dialog.js": "ui.dialog.css, ui.core.js, ui.widget.js, ui.button.js, ui.draggable.js, ui.mouse.js, 
ui.position.js, ui.resizable.js",
 "ui.core.js":null,
 "ui.widget.js":null,
 "ui.button.js": "ui.button.css, ui.core.js, ui.widget.js",
 "ui.draggable.js": "ui.core.js, ui.mouse.js, ui.widget.js",
 "ui.mouse.js": "ui.widget.js",
 "ui.position.js":null,
 "ui.resizable.js":"ui.resizable.css, ui.core.js, ui.mouse.js, ui.widget.js",
 //ui.redmond.css is my favorite skin for jQuery ui, :)
 "dialog.module" : "ui.core.css, ui.redmond.css, ui.dialog.js"
} );

Now we need to solve the second problem, resolving the url of resource, and we can use the following

matrix.url( {
 "ui.core.css": "js/jquery.ui/css/base/jquery.ui.core.css",
 "ui.dialog.css": "js/jquery.ui/css/base/jquery.ui.dialog.css",
 "ui.resizable.css": "js/jquery.ui/css/base/jquery.ui.resizable.css",
 "ui.button.css": "js/jquery.ui/css/base/jquery.ui.button.css",
 "ui.smoothness.css": "js/jquery.ui/css/smoothness/jquery.ui.theme.css",
 "ui.redmond.css": "js/jquery.ui/css/redmond/jquery.ui.theme.css",
 "ui.dialog.js":"js/jquery.ui/jquery.ui.dialog.min.js",
 "ui.core.js":"js/jquery.ui/jquery.ui.core.min.js",
 "ui.widget.js":"js/jquery.ui/jquery.ui.widget.min.js",
 "ui.button.js":"js/jquery.ui/jquery.ui.button.min.js",
 "ui.draggable.js":"js/jquery.ui/jquery.ui.draggable.min.js",
 "ui.mouse.js":"js/jquery.ui/jquery.ui.mouse.min.js",
 "ui.position.js":"js/jquery.ui/jquery.ui.position.min.js",
 "ui.resizable.js":"js/jquery.ui/jquery.ui.resizable.min.js"
} );

matrix( "dialog.module", function () {
 $( "#linkDialog" ).click( function() {
  $( "<h1>hello</h1>" ).dialog();
  return false;
 } );
} );

Let's how the resources are actually process.

So far so good, the fetching is parallel, and the evaluation is serial and in the correct order of dependencies. But the above configuration is too tedious. But if we want to use accordion widget, we need to repeat this process, obviously it violate the principle "Don't Repeat Yourself". Let's do "Convention over configuration". As I mentioned before, resource is extensible, you can extend handling of resource by adding a new resource type, which map to a new handler. For example, and x.mytype is mapped to a handler matrix.handlers.mytype. Here is methods that a handler can implement.

  1. load(resourceKey) [must implement]

    This method is to fetch and evaluate the resource and its dependencies into memory.

  2. release(resourceKey) [optional]

    This method is to release single resource (not its dependencies)

  3. url(resourceKey) [optional]

    This method is to build a url using a naming convention

  4. parseDepends(sourceCode) [optional]

    how to parse source code to build dependencies

Only the load method need to be implemented, other methods are optional. In the case here however, we want to reuse the default load method of javascript handler and css handler, all we need to do use extend url, and parseDepends. The following shows how to use the addHandler method


// it return back the new handler
//newHandlerName is also the name of new resource type
//matrix.addHandler(newHandlerName, baseHandlerName [,newBehavior])

matrix.addHandler( "uicss", "css", {
 url: function ( resourceKey ) {
  return matrix.fullUrl( matrix.baseUrl + "jquery.ui/css/base/jquery.ui." + 
matrix.resourceName( resourceKey ) + ".css" );
 }
} );

matrix.addHandler( "uitheme", "css", {
 url: function ( resourceKey ) {
  return matrix.fullUrl( matrix.baseUrl +  "jquery.ui/css/" + 
matrix.resourceName( resourceKey ) + "/jquery.ui.theme.css" );
 }
} );

var rdependencies = /^\s*\/\*[\w\W]*Depends:([\w\W]+)\*\//,
 ruiModule = /jquery\.ui\.(\w+?)\.(js|css)/gi;

//extend js handler
matrix.addHandler( "uijs", "js", {

 url: function ( resourceKey ) {
  return matrix.fullUrl( matrix.baseUrl +  "jquery.ui/jquery.ui." + 
matrix.resourceName( resourceKey ) + ".min.js" );
 },

 parseDepends: function ( sourceCode ) {
  var dependencies = [],
   module,
   dependText = rdependencies.exec( sourceCode );


  if ( dependText = dependText && dependText[1] ) {
   //dependText is something like
/*
   * jquery.ui.tabs.css
   * jquery.ui.core.js
   * jquery.ui.widget.js
*/
   while ( module = ruiModule.exec( dependText ) ) {
    //find jquery.ui.xxx.css and convert it to xxx.uicss
    //find jquery.ui.xxx.js and convert it to xxx.uijs
    
    // "xx" + ".ui" + "js" == "core.uijs"
    // "xx" + ".ui" + "css" == "core.uicss"
    dependencies.push( module[1] + ".ui" + module[2] );
   }
   return dependencies.length ? dependencies.toString() : null;
  }

  return null;
 }
} );

In the above code, the url method is straight forward, it is used to map a resource key to a url, since jQuery UI team follow a naming convention of their components, we can use this, and use a function to calculate the url based on resource key. What is interesting this the parseDepends method. The idea is that if we can parse it by human eye, we can parse it by code. The method basically extract the headers we modified before, and convert them into resource keys, and then build dependencies. With all this code in place, we can throw away the previous dependencies and url configuration. Let's write a bit of code, to register the dependencies and the locations of all jQuery ui files.

matrix.baseUrl = "js/";
matrix.debug();

matrix.depend( {
 "uicss.module": "core.uicss, redmond.uitheme"
} );

$.each( 
("accordion,autocomplete,button,datepicker,dialog," + 
"progressbar,resizable,selectable,slider,tabs," + 
"draggable,droppable,mouse,position,sortable").split( "," ),
 function (index, value) {
  var resourceKey = value + ".module";
  var depends = value + ".uijs, uicss.module";
  matrix.depend(resourceKey, depends);
 } );

I agree that I need to spend more time to do this kind of convention over configuration. But I think it is worthy, because I can dynamically load the the resources of just of my widget, any widget in jQuery UI, not just dialog. In the next post, I will explain dependencies refresh, release release, performance and other advanced topic in using matrix.

Apr 3, 2011

tweaking HTTP header to imporve performance

Lately, I am doing some client side optimization. One of the techniques is cache. When browser firstly request a resource and get the response, the cache subsystem of the browser need to determine whether to cache resource. It is based on the response header Cache-Control, and Expires. When neither of them exists in the response header, browsers use their own strategy to cache, these strategies vary from browser to browser, so they are not predictable, we should explicitly set these headers. To explicitly disable cache in browser, we can use the following

Pragma: no-cache 
or
Cache-Control: no-cache 
or
Cache-Control: no-store

The Pragma: no-cache header is included in HTTP 1.1 for backward compatibility with HTTP 1.0+. It is technically valid and defined only for HTTP requests; however, it is widely used as an extension header for both HTTP 1.0 and 1.1 requests and responses. HTTP 1.1 applications should use Cache-Control: no-cache, except when dealing with HTTP 1.0 applications, which understand only Pragma: no-cache. According RFC 2616, Cache-Control: no-cahce is semantically different from Cache-Control: no-store. non-cache still allows browser cache a response, and the cache needs to re-validate the response with the origin server before serving it. no-store request browser not to cache response at all. However most browser treat "no-cache" as "no-store".
To implement the semantics of "no-cache", we should use Cache-Control: max-age=0 or Cache-Control: s-maxage=0 or Cache-Control: must-revalidate.

To explicitly to cache resource, we should use the following.

Cache-Control: max-age=3600
Cache-Control: s-maxage=3600
//or
Expires: Fri, 05 Jul 2002, 05:00:00 GMT

The Cache-Control: max-age header indicates the number of seconds since it came from the server for which a document can be considered fresh. There is also an s-maxage header (note the absence of a hyphen in "maxage") that acts like max-age but applies only to shared (public) caches. The deprecated Expires header specifies an actual expiration date instead of a time in seconds. The HTTP designers later decided that, because many servers have unsynchronized or incorrect clocks, it would be better to represent expiration in elapsed seconds, rather than absolute time. An analogous freshness lifetime can be calculated by computing the number of seconds difference between the expires value and the date value.

If browser hit the same url again, browser will firstly ask the cache sub-system to get a cache copy. If the cache is still fresh based on "Cache-Control" or "Expires" header, the cache is returned, and no request is sent to server. If the cache expired, determined whether a validator was send from the previous response, different request will be sent. If a validator of previous response is "Last-Modified", the following will apply.

//validator of previous response
Last-Modified: Sun, 03 Apr 2011 14:34:43 GMT

//validator of next request
If-Modified-Since: Sun, 03 Apr 2011 14:34:43 GMT

If a validator of previous response is "Etag", the following will apply

//validator of previous response
ETag: 100

//validator of next request
If-None-Match: 100

Normally we should use either "Last-Modified" or "ETag" in response from server. If neither of them is available in the previous response, then a new request will make. You should note that validator is independent with "Cache-Control" and "Expires" header, which means it can be used without these two header.

In asp.net, if you don't do anything, the following default header will be generated.

HTTP/1.1 200 OK
Server: ASP.NET Development Server/10.0.0.0
Date: Sun, 03 Apr 2011 14:47:30 GMT
X-AspNet-Version: 4.0.30319
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 509
Connection: Close

Because the Cache-Control header does not specify a max-age, by default is zero, and because there is no validator available, is semantically "Cache-Control: no-store". So we don't need to do anything if we want the client not to cache the response, because it is by default. The following is the default header of next response.

GET http://localhost:1491/WebSite1/Cache.aspx HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Accept-Language: en-US
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Host: localhost:1491
Cookie: ASP.NET_SessionId=imdo5jrzxkpw2ohy24guaw0b

The following is a typical header to enable client side cache.

Cache-Control: max-age=3600
Last-Modified: Sun, 03 Apr 2011 14:34:43 GMT

or 
Cache-Control: max-age=3600
ETag: 100

We should prefer "Cache-Control" over "Expires" because it is smarter. When the resource is static, we should config web server to add a Last-Modified header but no using ETag, because it simply and easier. And ETag is problematic in IIS web farm. When the resource is dynamic, we should use dynamically generate an ETag as hash to compare change. Here is some server side code.

private void AddCacheControlHeader(int seconds)
    {
        Response.AddHeader("Cache-Control", string.Format("max-age={0}", seconds));
    }

    private void AddETagValidator()
    {
        string hash = "xyz"; //calcuate a hash match the request
        Response.AddHeader("ETag", hash);
    }

    private void ProgessETagValidator()
    {
        string hash = "xyz"; //calcuate a hash match the request
        if (Request.Headers["If-None-Match"] == hash)
        {
            Response.StatusCode = 304;
            Response.End();
        }
    }