Dec 23, 2011

Why dynamic resource loading is so important for web application?

When I finished matrix.js 0.1, I asked for feedback from my friends. But the general response seems that I am trying to solve a problem which does not exist, so the library is useless. But I am not disappointed. I make it not because we have a pain point in my current projects, but because I see it is the way that we need to build future web application. It is better to explain what the problem is as I see, before I propose on a solution. I believe that, as we are developing web application, dynamic resource loading will be a essential function. The current way to import resource using <link> <script> tag statically and tools like combres will not be able to meet the needs of building web application.

The future web application? Isn't that we are a building web application. Yes, in a way we all building web application. There are two extreme of web application. On one extreme, web application completely depends of the <form> tags, all interaction depends on form postback, and page get re-rendered, one example are government web sites. On other extreme, web application does not have <form> tag, all interaction depends on ajax call, all UI is generated by JavaScript, template and the application stay in the same url until it is closed by user. This kind of application is fast and fluid, is as responsive as native desktop application. Gmail is an example close to this extreme. We are normally building web applications somewhere in between these two extreme.

To increase the loading speed of website, Minimize HTTP Requests is one of the rules listed in "Best Practise for Speeding Up Your Web Site". In .net, to minimize http requests, a popular approach is use tools like combres to combine, minified, gzip a set of resource, configure response header for client side caching. So that one big JavaScript file and one css file are generated and cached at client side. When we are building our current applications, the resource like JavaScript, css can be known more or less in advance. Because all pages in the website pretty much share some library like jQuery, a couple of widget plugin, like calendar, accordion, auto complete and so on. We can hand pick a set of resources required in specific page, or for the whole website. If we have a page require very different set of resources, we may need to pick another set of resources targeted for that page. So far so good.

As we are moving to an other extreme, building web application like Gmail, or even more complex, the architecture of our application changes. In this extreme, the application is responsive as desktop. The application compose and aggregate lots UI modules, or views, each views has their unique resource, and in a lot of case, we cannot anticipate which views will be used by user. All the views are built on demand of user. We need to load lots of client side template, view specific JavaScript, and unload them when view is not used, because the application is open for 8 hours a day continuously, memory management is critical for good client side performance. These requirements can not be solved by combres or statically resource loading. Combres can shorten the time to transfer resource from the server to browser. But it can not solve our architecture requirement like client side memory management, reusable, modular component design, dependencies tracking among resources. This resource is like Dynamic-link library, but it just happens at browser. Browser is our VM, not CLR not JVM. In .net, CLR is the run time. Assembly(DLL or EXE) is reusable unit. The loading of assembly is highly sophisticated. Here are some features.

  • Assembly Manifest contains of dependencies of dll/exe
  • Appdomain
  • Security check
  • Late binding
  • Dynamic loading

matrix.js is my starting point of building a true web application. I try to address similar issues we have like in assembly loading in .net, but I just do it for the browser vm not for CLR or JVM. Unlike CLR, browser vm has some advantage CLR does not have. In CLR, you can release your DLL, until you kill your AppDomain, in browser, you can unload everything dyanamically. You can remove DOM elements, you can unload a library , like jQuery by calling "delete window.jQuery", you can unload css by remove the link tag.

As we move to web application extreme more and more, preload two resource files (one js, one css) is like we load one gigantic EXE for a win32 application. Not only it is bad from the aspect of memory managemeent, it is even worse from the aspect of software engineering. It become monolithic artifact, while we want to have modular, reusable component.As HTML5 is ready, we will be building more and more application like Gmail and more complex, I believe dynamic resource loading is essential but not a fancy feature. Maybe you don't see the need to build a true web application right now, if you have, you can check out matrix.js

Dec 7, 2011

What does the size of a JavaScript Library say about it?

When people introduce a JavaScript library, they often mention that the size of the library is very small, like "it is small and light weight, xx K minified, in fact is about yy k when gzipped". Does this means a library runs fast or use less memory? Neither. We know that 1k size virus can use up all your machine resource, and make it dead slow, right? What it means the library can be transported to client side faster. Normally the more features a library provide, the bigger the size. You can use profiler (like the one built in Chrome) to collect data when using the library in real life scenario.

Sep 22, 2011

The design principle of viaProxy

[update: the library is now in github, and the sample code discussed in this post is here]

viaProxy is a client side JavaScript library that can be used to build complex, fast and fluid web UI yet in a manageable code complexity. It is a set of low-level api which is built on top of jQuery and enable you to synchronize your view and model using imperative programming (code only) or declarative programming ( mark-up only) or both. You can use it to write testable views, modules, plugins, and aggregate them into complex view.

About a year ago, that is after I wrote jquery-matirx, I had been thinking a problem in a project. We were developing a rich web UI using JavaScript intensively , but as the logic of application grows, the JavaScript code explode, and become buggy and spaghetti like, as our logic is distributed every where in the handler. Although, I create a solution to dynamically load view into the page at client side with some template, but does not solve the issue completely. So I think, is there a UI framework to address this issue systemically. I studied libraries like Backbone.js and knockoutjs , which seems to be good UI framework. Initially, I thought I had sufficient JavaScript competency to use them, it turned out I had lots of difficulties to write a "hello world" app with them, and even greater difficulties in solving a real world scenario. The syntax and convention are just too cryptic for me, and it is hard for me to do unit test and debug. So I decided to write my own, that is how I started viaProxy

Before I discuss how to use the library, I want to put the design principle of the library upfront, which is fundamentally important to use the library.

The principle is to separate model interaction from model presentation.

You can think of model interaction as CRUD(CREATE, READ, UPDATE, READ), and model presentation as view rendering. A view is the typically the UI that user can see and feel, a model is typically business object of your application. I want to use the word "typically", because sometimes, the difference between view and model can be blurred. A view can be just another model, we will discuss this in the future post.

Let's take a look at "Hello world" application to understand how model and view play together. The application allows user enter a name, and click a button, the application display a greeting message. Here is markup that defined the view.

<label>Please input your name: <input type="text" id="txtName"/>
 <input id="btnGet" value="Get message" type="button"/>

<div id="divMessage"/>

And here is a piece of javascript to implement the user story.

$(function () {                                       
 $("#btnGet").click(function () {     
  var name = $("#txtName").val();                     
  if (name) {                                         
   $("#divMessage").text("Hello," + name);            
  } else {                                            

You might be laughing at how stupid the code it is. But there is nothing wrong with it, in fact, it is simple and elegant, if this is everything that the user want from the app. However what user want is far more complex web application. If you have a little bit web UI programming experience, you probably agree that, by applying this style of implementation to develop complex UI, the code will soon explode and become spaghetti.

Let's take a closer look. In the click handler, it by-pass the model and directly update the view, in fact it does not even define a model, it just directly convert user's click action into updating view. When user stories grows, we need to create more viewsto implement them, and we need more view handlers to take care of user interaction. Our view handler will have hundreds or thousands lines code doing thing like "If user input this, do this, if input that do that, because of this change, we also update an this view, because that change, we also need to update that." Very soon, the code complexity will grow out of control. So what is wrong? The fundamental fault is mixing model interaction with model presentation, because our view handler has too much thing to worry.

Let's refactor the above code using viaProxy. At first sight, the refactory may require you to write more code, what I show is how viaProxy works under the hood, I will introduce higher level viaProxy API which can reduce your code to minimum or no code at all. The first step is to define a model, which is missing from the previous implementation. Without model, we can not possibly separate a model interaction from model presentation. Model is center of an app. So here is the model.

var rootProxy = via();            
var helloAppModel = {                                     
   name: null,                                                        
   greeting: "Hello",                                                 
   message: function() {                                              
     return ? this.greeting + "," + : "";          
rootProxy.insert( "helloApp",  helloAppModel);

The model is a javascript object which has 3 members name, greeting, and message. The message function combines greeting and message. It is very pure and simple, that it does not inherit from anything other than object, it does not use hard-to-understand concepts like "Observerable", "viewModel", which are used in other framework. As your can see, we put our model into a repository using a proxy with a namespace "helloApp", this prevents you from direct access to the model, all access must be via the proxy, this is why the library is named viaProxy.

Now let's take on model interaction. Instead of using raw DOM event directly, we wrap it into higher abstraction, view event. View event can be one-to-one mapped to DOM event, but it can be customized event. For example, we can create an "enter" event which will trigger when "keypress and keycode === 13" event triggers, I will cover that later. Here is how we attach a view handler to the view event, like below.

$( "#btnGet" ).addViewHandler( "click", "", 
  function( viewContext ) {
    var value = $( "#txtName" ).val();  
    viewContext.updateModel( value );
} );

The view handler is also associated with a path("") which is pointer to the model in repository. In the view handler, it does only one thing, updating the model with the user's input, however it does not care about updating the divMessage, because this is the job of model handler. This is a big difference from the previous implementation. Remember, model handler is only place do model presentation. Here is the model handler to update the divMessage.

$( "#divMessage" ).addModelHandler( "helloApp.message", "afterUpdate", 
   function ( modelContext ) {
     var value = modelContext.currentValue();
     //var value = via("helloApp.message").get();
     //"this" refer the divMessage
     $( this ).text( value);
} );

The semantics here is when model event "afterUpdate" happens to "helloApp.message" in the repository, call this handler to update view "divMessage". In the model handler, it also does only one thing, update the view. You maybe notice that the previous view handler update "name" property, why the event of "afterUpdate" is raised for "message" property, this is because viaProxy knows this dependencies between "name" and "message". Here you don't need to write some thing like "observerable()" which is used by other library. After creating the model, adding a viewHandler, a modelHandler, your "hello world" application is up and running.

So what is the big deal of separation model interaction from model presentation. It might not be obvious for you right now. But here is a few things that I can think of. Firstly the separation can reduce code complexities to be manageable. As your application complexities grow, your code will grow linearly but not exponentially. You can add more view handlers to a view event to update more model, and add more model handler to model handler to update more views.

Secondly, the separation make each handler focus on one thing, so that code is more reliable, more testable. Yes, we can unit test our UI. I will cover that later. The idea is that we can test our view handler by triggering fake mouse event without a real mouse click. Because view is rendered by model handler, to test a view is to test model handler. We can test model handler by updating our model using via proxy.

Thirdly, you have a explicit, complete model, a single copy of logic, not a duplicated partial model distributed in spaghetti code. And the model can scale, you can add more model into different name space. The model the king of your UI, which rules the application. Different parts of model become shareable, and connected. The code become more easy to understand because of this.

There are other creative uses of viaProxy. For example, in our ajax callback, instead of update the UI directly, ajax callback update model using viaProxy, and your UI will be updated indirectly in a model handler. Instead of making ajax call directly in your view handler, your view handler can update model using proxy, that will trigger model handler, which will make the ajax call.

Through the "hello world" example, we can see that the design principle of viaProxy is to separate model interaction from model presentation. Essentially, it provides three following mechanisms to facilitate the separation.

  1. A proxy to build and access models
  2. An extensible model event mechanism to connect model to view
  3. An extensible view event mechanism (based on jQuery event) to connect view to model

Now that we know the design principle behind viaProxy, what is next. There are still lots of challenge. Does it work in a real world scenario or in large scale implementation, does it work with other existing controls or plugins, what if my model is complex object like array instead of simple string or number, is it extensible, how about validation, does it encapsulate too much or too little, is it too automatic, how easy is it to customize it, can I continue to use my programming style while using viaProxy, what about learning curve? I have considered all these when I develop the library, and I will discuss more in the future posts. Let me a comment, tell me what you think and stay tuned.

Jul 8, 2011

f# essential - part 14 - object oriented without class

JavaScript is duck type language, if you expect a object has a method, you just call object.method. It supports Polymorphism very well, but you don't need implementation inheritance, and you don't need prototype, constructor at all. F#'s record type has similar concept. It is not a class, but it is strongly type, and it is super easy to create, it is like javascript object in strongly type. I think it is the one of best features in F#. Its syntax is as following

type YourRecordType = 
    { member1: type1;
      member2: type2 }

      member x.function1() = 

Why it is so interested to me. So first, the definition of record type is similar a interface. So it is strongly type. However, the implementation is not defined. More importantly, it is very easy to create an object, which is the implementation. For example

let yourObject = {
      member1: m1;
      member2: m2;

Woo, that is so simple, if you make a member mutable, you can even switch implementation on the fly.

f# essential - part 13 - pattern matching over .net type

f# use unique syntax to match type like the following,

let divide1 x y =
      Some (x / y)
      | :? System.DivideByZeroException -> printfn "Division by zero!"; None

let result1 = divide1 100 0

let recognizeType (item : obj) = 
    match item with 
    | :? System.Int32 -> printfn "An integer" 
    | :? System.Double -> printfn "A double" 
    | :? System.String -> printfn "A string" 
    | _ -> printfn "Unknown type" 

f# essential - part 12 - function is not method, is not delegate, what is it anyway

As previous post discussed, f# function is not a method, because it support closure, it curry and so on, and .net method does not support these. It looks very much like a delegate (aka lambda, anonymous method) in .net. For example, the following looks like it is a delegate.

open System
open System.Linq

let l = [ 1; 2; 3 ]
let conditionFunction i = i > 1
let c = l.Where(conditionFunction).Count()

But it is not, the case compiler is actually something similar like the following.

let l = [ 1; 2; 3 ]
let conditionFunction i = i > 1
let conditionDelegate = new System.Func<_, _>(conditionFunction)
let c1 = l.Where(conditionDelegate).Count()

So let's use reflector again.

public static class Program
    // Methods
    public static bool conditionFunction(int i)
        return (i > 1);

    // Properties
    public static int c
            return $Program.c@36;

    public static Func<int, bool> conditionDelegate
            return $Program.conditionDelegate@34;

    public static FSharpList<int> l
            return $Program.l@32;

    // Nested Types
    [Serializable, CompilationMapping(SourceConstructFlags.Closure)]
    internal sealed class conditionDelegate@34
        // Methods
        internal bool Invoke(int arg)
            return Program.conditionFunction(arg);
 public static void main@()
        FSharpList<int> l = l@32 = FSharpList<int>.Cons(1, FSharpList<int>.Cons(2, FSharpList<int>.Cons(3, FSharpList<int>.get_Empty())));
        Func<int, bool> conditionDelegate = conditionDelegate@34 = new Func<int, bool>(new Program.conditionDelegate@34().Invoke);
        int c = c@36 = Program.l.Where<int>(Program.conditionDelegate).Count<int>();

f# essential - part 11 - ignore function

If you call a function which does not return unit type, you can not call it like the following.

let f1() = 
    //do some side effect stuff

You have to do the following

f1() |> ignore
let _ = f1()

f# essential - part 10 - why and when f# does not need null value,

I am surprise when I don't see a need to use null in f#. It use a generic record type "option", which is a very elegant solution to express nullness. In c#, we have reference type and value type, in .net 1/1.1, we express nullness for reference type using "null", in .net 2 above, we can use Nullable type of value type. However, f# unify this using option.

The null keyword is a valid keyword in the F# language, and you have to use it when you are working with .NET Framework APIs or other APIs that are written in another .NET language. The two situations in which you might need a null value are when you call a .NET API and pass a null value as an argument, and when you interpret the return value or an output parameter from a .NET method call. For more please refer to MSDN

f# essential - part 9 - no need to use ref record type in class

As I discussed it in post f# essential - part 5 - mutable, ref, ref record is used for closure. And mutable is similar concept of variable in c#. So that you can not use mutable defined outside of a function. So the following it is wrong

let build_t() = 
    let mutable x = 0
    let t() =
        x <- 1 + x
        printfn "you call me %i time(s)" x

let t =  build_t()

You have to convert it ref record type like the following.

let build_t() = 
    let x = ref 0
    let t() =
        x := 1 + !x
        printfn "you call me %i time(s)" !x


If we define method in a class using F#, and we want to update the member field in a method, should we also use mutable or ref record. It turns out, both of them are supported.

type Foo() = 

    let mutable callT1Count = 0
    let callT2Count = ref 0

    member x.t1() = 
        callT1Count <- callT1Count + 1
        printfn "you call t1 %i time(s)" callT1Count

    member x.t2() = 
        callT2Count := !callT2Count + 1
        printfn "you call t2 %i time(s)" !callT2Count

let f = new Foo()


I used reflector to decompile it to c# code as follow to know that that is implemented in .net

[Serializable, CompilationMapping(SourceConstructFlags.ObjectType)]
public class Foo
    // Fields
    internal int callT1Count = 0;
    internal FSharpRef<int> callT2Count = Operators.Ref<int>(0);

    // Methods
    public void t1()
        FSharpFunc<int, Unit> func = ExtraTopLevelOperators.PrintFormatLine<FSharpFunc<int, Unit>>(new PrintfFormat<FSharpFunc<int, Unit>, TextWriter, Unit, Unit, int>("you call t1 %i time(s)"));
        int num = this.callT1Count;

    public void t2()
        Operators.op_ColonEquals<int>(this.callT2Count, Operators.op_Dereference<int>(this.callT2Count) + 1);
        FSharpFunc<int, Unit> func = ExtraTopLevelOperators.PrintFormatLine<FSharpFunc<int, Unit>>(new PrintfFormat<FSharpFunc<int, Unit>, TextWriter, Unit, Unit, int>("you call t2 %i time(s)"));
        int num = Operators.op_Dereference<int>(this.callT2Count);

Why mutable is not supported in closure? I think the reason is that when you create an instance of Class, the variable (mutable) is already closed in an instance. Of course, ref type is still supported, but but I think it is double closed. We only need to use variable(mutable) in class.

f# essential - part 8 - talking to c# library

When I found that syntax in f# to support curry, I am very excited. May be we can do the same thing with .net library written in c#. For example, maybe I can use Console.WriteLine as a function to write the following code.

let greet = Console.WriteLine "hello, {0}"
greet "fred"
Console.WriteLine "hello, {0}" "fred"

The compiler give me an error "The value is not a function and can not be applied". So F# function is different from c# method. To call this method, you have to use a tuple syntax. But you can wrap a method easily into a f# function, then you can do curry

Console.WriteLine ("hello, {0}", "fred")

//wrap to a f# function
let writeline message (name: string) = 
    Console.WriteLine(message, name)

let greet = writeline "hello, {0}"
greet "fred"

Jul 5, 2011

f# essential - part 7 Curry

Function application, also sometimes referred to as function composition or composing functions is a big thing in functional programming. One style of composition is called Curry. Curry is feature of lots functional language. When you have a function with two parameter, you can pass one argument to the function, that create an new partial function, which accept one parameter. This partial function is also called curried function. F# support this in a very elegant syntax.

let print x y = 
     printfn "%i" x 
     printfn "%s" y 
let print100y = print 100 //create a curried function 
print100y "hello" //call the curried function 

If you want to fill in the second argument to create a new function, this is also support, like below.

let printx_hello x = dosomething x "hello" 
printx_hello 100

Jul 4, 2011

f# essential - part 6 - tuple

In other language, a method can not return more than one parameter. There are two way to get around this, use passed-by-reference parameter, or use composite type to encapsulate two or more fields in to one object. Both of them are not elegant. However, f# can express this using a very simple syntax to solve this problem, "tuple", which is a strongly typed structure. Here is an example.

let (x, y, z) = (1, 2, 3)
let x, y, z = 1, 2, 3

f# essential - part 5 - mutable, ref

As mentioned in previous post, the "let" syntax define a identifier binding, which is immutable. So it is not variable in the sense of traditional programming. But we can not programming in pure functional language environment. We need to create state, and side effect. F# is pragmatic and play nice with .net framework, so it has to be support variable. To use variable, you just need to add a "mutable" in front identifier, then the identifier became variable in the sense of traditional programming.

let totalArray2 () =
    let array = [|1; 2; 3|]
    let mutable total = 0
    for x in array do
        total <- total + x

    printfn "total: %i" total


However, mutable does not support closure. So if you write the following code, you will get a compiler error.

let totalArray2 () =
    let array = [|1; 2; 3|]
    let mutable total = 0
    let sum() = 
        for x in array do
            total <- total + x

    printfn "total: %i" total

//The mutable variable 'total' is used in an invalid way. Mutable variables cannot be captured
//by closures. Consider eliminating this use of mutation or using a heap-allocated mutable 
//reference cell via 'ref' and '!'. 

The reason is that closure is bound to identifier but not variable. This identifier need to be immutable. To get around this, we can use immutable record identifier, but a member is mutable.

type MyRefRec<'a> = { mutable myref: 'a }
let totalArray1 () =
    let array = [|1; 2; 3|]
    let total = { myref = 0 }

    let sum() =
        for x in array do
            //now total is not a mutable type, so it can 
            //be captured by closure
            total.myref <- total.myref + x


    printfn "total: %i" total.myref

You can see, MyRefRec is generic record. It has only one field. F# provide a shortcut and operator to simplified the above steps. This is "ref". So the above code can not be simplified as below.

let totalArray4 () =
    let array = [|1; 2; 3|]
    let total = ref 0

    let sum() =
        for x in array do
            //now total is ref type 
            total := !total + x


    printfn "total: %i" !total

f# essential part 4 - closure

f# support closure. Here is the definition of closure

In computer science, a closure (also lexical closure, function closure or function value) is a function together with a referencing environment for the nonlocal names (free variables) of that function.

In f#, you can define functions within other functions. The inner function can use any identifier in scope, including identifiers defined in outer function. When the inner identifier return as result of the outer function, identifiers defined in outer function is still being captured by the inner function, the inner is called closure.

let buildGreet greeting  = 
    let prefix = Printf.sprintf "%s ," greeting
    let greet name = 
        Printf.sprintf "%s%s" prefix name
    greet //greet is a closure, even the it is return it can still access,

let getHelloMessage = buildGreet "hello"
let message = getHelloMessage "fred"
printfn "%s" message

Jun 29, 2011

f# essential - part 3 matching shortcut

F# provide a shortcut to define a function which purely use input parameter for matching purpose. For example the following are equal.

let printInt =
    function 1 -> printfn "1"
            | 2 -> printfn "2"
            | _ -> printfn "something else"

let printInt number =
    match number with 1 -> printfn "1"
                      | 2 -> printfn "2"
                      | _ -> printfn "something else"

let printInt number =
    match number with
        | 1 -> printfn "1"
        | 2 -> printfn "2"
        | _ -> printfn "something else"

f# essential - part 2 - pipleline operator

Before I discuss pipeline operator, let's talk about operator. F# allow us to define operator as function. The following are infix operators

$ % & * + - . / < = > @ ^ | 

The following are prefix operators.

! ? ~

You can define a infix operator like below

let (operator) ops1 ops2 = ...
//for example 
let (+) a b = a - b

You can define a prefix operator like below

let (!) a = 1

So you can change your expression from "add 2 4" to "2 + 3" or "m1 a" to "!a", let's defined our pipleine operator as below.

let (-->) x f = f x
let rb = 0.5 --> System.Math.Cos

However, F# already define the "|>" for us, so we should use "|>". Operator is not a function, definition, rather you should think of it as macro. The compiler will change the syntax before it compile.

f# essential - part 1 binding

I was very puzzled with some concept when learning F#. Here is some. Even I explain it here, you may be still confused, but it is ok, as you read on.

let t1 = 1
//fsi shows val t1 : int = 1
//think of it as (val t1) bind to (int = 1)

Based on the concept of other language, you think t1 is variable, and it is initialized or assigned with value 1. But here it should be understood as identifier t1 is bind to a value, the value is not function value, it is int 1. So what is the big deal of "identifier", "bind", "function value", and "value", we will see later. If you run the following expression in fsi

//fsi shows val it: int =1

This is evaluation of expression "t1", the process is to bind value int 1 to unknown identifier. Let's read another example.

let t2() = 1
//fsi shows  val t2 : unit -> int
//here fsi should shows
//val t2: unit -> int = <fun:it@91-2>

Here it means identifier t2 is bind to a value, the value is a function value, and the value is a function take unit type parameter and return a int type value. Here unit is similar the concept of "void" in other language. How do I prove my remark "here fsi should shows...", because we can find out what is the value of t2, by evaluating expression "t2" like below, it return the value of t2, here what I really means "the value that t2 is bind to "

//val it : (unit -> int) = <fun:it@91-2>

t2 is different from t1 in that the value t2 bind to is a function, while the value that t1 bind to is not. Because the value is function, we can call the function like below. The concept of "function is a value" is an important concept in functional language, it may not looks a big deal for you. But it is very powerful. Please note the function here is not a traditional function in VB, a method in c#, conceptually it is like C#'s delegate, or javascript's function, people call it lamda in other functional language.

//fsi shows val it : int = 1

The type description generated by fsi need some explanation.

int //int type
int -> string //function type take one int input, return one string
int -> string -> double // a function take one int, one string, return one double
int * string -> double // a function take a (int, string) tuple , and return one double

May 6, 2011

Semantics of undefined and null in javascript

Recently, I was discussing with a friend about the difference between undefined and null in javascript. I am surprised that I gave an explanation that he can understand, more surprisingly, I can understand too. Sometimes, people give an explanation, which they don't understand themselves, and also confuse others. Before going further, here is the explanation from JavaScript: The Definitive Guide

You might consider undefined to represent a system-level, unexpected, or error-like absence of value and null to represent program-level, normal, or expected absence of value. If you need to assign one of these values to a variable or property or pass one of these values to a function, null is almost always the right choice.

The explanation is confusing to me. What does this means in my daily coding, in what scenario I must use one but not the other or a scenario I can use either of them? So if null is almost the right choice, but why we have "undefined", is there any technical reason or semantic reason? Let's see why we have "undefined" value in javascript from technical perspective. When we write the following code, I can say, I declare two variables and they are not assigned with any value, and the default value of each is undefined.

var x;
var loooooooooooooooooooooooooong;

After reading the article Understand delete, I understand that, javascript variable mechanism. When you declared variable like above, you add a key/value pair entry to a mysterious object, VariableObject, which is dictionary. Here is pseudo code, that the engine will convert to

VariableObject["x"] = undefined;
VariableObject["loooooooooooooooooooooooooong"] = undefined;
//semantically, it means the following
//VariableObject.add("x", undefined);
//VariableObject.add("loooooooooooooooooooooooooong", undefined);

If the above example, two key/value pairs are added to the VariableObject dictionary. So both the key and value consume memory, so if you have a longer variable name, regardless its value, it use more memory than short variable. This is different from c++. In c++, a variable name is nick name of memory address. So practically, we should use shorter variable name, or use minifier to rename your variable. The VariableObject is special in that it is created by runtime. If the code is run in Global scope, the VariableObject is accessible as window. If it is run function scope, it is not accessible at all, which is known as Activation Object. Supposed it is run in global scope, it is same as the following.

window["x"] = undefined;
window["loooooooooooooooooooooooooong"] = undefined;

In the above case, variable x is said declared because its key is in the dictiobary, but its value is undefined. If the key is not even in the dictionary, then it is undeclared. Technically, there is difference between "undeclared" and "declared but undefined". But the following undefined check does not tell the difference.

//suggested by jQuery Code Style Guildeline
//undefined check
//Global Variables: 
typeof variable === "undefined"
//Local Variables: 
variable === undefined
object.prop === undefined

If you really need to know the difference, you need to use catch, because accessing undeclared variable directly will throw an exception.

function test(variableName) {
  try {
     var temp = eval(variableName);
    if (temp === undefined) {
       return "\"" + variableName + "\" is declared, its value is undefined"; 
    } else {
       return "\"" + variableName + "\" is declared, its value is not undefined"; 
  } catch (e) {
      return "\"" + variableName + "\" is undeclared";

var y;
var z = null;

alert(test("x")); // "x" is undeclared
alert(test("y")); // "y" is declared, its value is undefined
alert(test("z")); // "z" is declared, its value is not undefined

Most of time, we don't care the difference between undeclared and undefined. Practically, we can treat it the same, if a value is undefined, it does not exist in dictionary, although it is not quite true. If we accept this, we can use the undefined check as the jQuery code style guidline recommend.

Back to the question, why we need to have undefined? This is because variable is key/value entry in dictionary, this is because we can add entry into dictionary in runtime. If its value is undefined, practically it does not exist in the dictionary. Using null simply simply can express this semantics, because its value is "null", it is already in the dictionary. Now we know the techinicall difference, how can apply them into our coding. undefined check is normally used, before defining it. Here is an sample

//if somebody defined, if it has been defined, regardless its value,
//don't define it again.   
if (console.log === undefined ) {
    console.log = function () { ... }

function css(key, value) {
  //if user does not give a value,
  //he want to get the value
  if (value === undefined) {
     return db[key];
  } else {
    //otherwise user want to set the value
   db[key] = value;

But what about null in javascript? We know how technically it is different from undefined. What is its semantics? Short answer is it depends. It is up to you how to interpret it, and only you can define it in your application. In my matrix library, I use null to represent the case when no a resource has no dependencies, the undefined value to represetn the case when dependencies is yet to know, the semantics is quite different. But its semantics can be others if you want.

if (depedencies["x.js"] === undefined ) {
       //go figure out the what dependencies is and come back later

    } else if (depedncies["x.js"] === null) { 
       //there is no dependencies, load it directly.

    } else {
        //load dependencies["x.js"] first, because it is not empty.

Apr 30, 2011

matrix.js introduction

Essentially, matrix.js is like other script loader, but it does other things such extensible resource type, dependencies registration and reload, resource locations, resource release and others. In the following, I will explain what the library can do. All these example presented here can be found at GitHub

The most used method in matrix is the "matrix" method. Here is the definition.

//returns a promise
//resourceKeys is a string or an array, e.g "a.css, b.css" 
//or ["a.css", "b.css"] or ["c.js", "a.css, b.css"]
matrix(resourceKeys[, loadByOrder])

If you have no configuration, the resource key is is treated as url of the resource. Resource key follow the format of [resourceName].[resourceType]. There are three build-in resource handler in matrix, "module", "js", and "css", resourceKeys can be a string or array. Most of the time, you should use string format resourceKeys. If it is a string, this means resource are fetched in parallel, if it is an array, this means resources are fetched in serials, most of time, we use resourceKeys as string. I will discuss more about it later. Just like any other script loader, you can load javascript and css like the following.

var promise = 
matrix("/css/reset.css, /css/base.css, /js/utilities.js, /js/app.js", 

Behind the scene, the loader will fetch two css and two javascript files using ajax call in parallel, and evaluate each source file into the page when it is ready at client side. and run fn1, fn2, fn3 after all resources are loaded. It is important to know that, the order of making ajax call is not necessary the order when ajax call is finished, so the order of evaluation is not guaranteed. If the order of evaluation is important, you can use the following code.

var promise = 
matrix("/css/reset.css, /css/base.css, /js/utilities.js, /js/app.js", 

Behind scene, the load method actually convert it into the following call.

//reset depends nothing
matrix.depend("/css/reset.css", null); 

//base.css depends on reset.css
matrix.depend("/css/base.css", "/css/reset.css"); 

//utitlies.css depends on base.css
matrix.depend("/js/utilities.css", "/css/base.css");  

//app.css depends on utilities.css
matrix.depend("js/app.css", "/js/utilities.css");  
var promise = matrix("/js/app.js", true).done(fn);

Dependencies between the module is important in resource loading, sometimes is more complicated than serialized dependencies like above case. When loading resource, matrix follows a couple steps. Firstly, matrix determines the type of resource and the correct handler for processing the resource request. After that, matrix use the handler to execute three tasks(fetch, parse and evaluation). Fetch is to get the content ready at the client side. Parsing is an optional step, matrix currently extract dependencies from the content in this step, and this is necessary only when dependencies information is not pre-configured, and parsing method is implemented by the handler. Evaluation is to integrate the resource into the page. These three tasks are implemented by a resource handler.

Let's try a more complicated case. jQuery UI library offer lots of widgets. It is a highly modularized library, some components(like ui.mouse.js) are reused over the library like. If you only want to use a certain widget,e.g dialog, you can go to their home page, deselect all, and just select dialog, and it create a a package that include just the css, js file that dialog widget use. This is a great feature. But there is still a problem. If I have two page, page one need a,b,c widget, and page two need b,c,d widget, so I need to create two js package. To save the trouble, I might better off just create a package file that include all the modules. However this might load too many modules that I don't need. We can use matrix to load the minimum set of resource to be able to use a widget. First I will show you the hard way, later I will show you easy way. But regardless of which, we need to solve two problem, resource dependencies and and resoure locations.

To understand dependencies in jQuery UI, you need to download all the source of jQury UI. Open the source file in jQuery.ui.dialog.min.js, the header says

 * jQuery UI Dialog 1.8.10
 * Copyright 2011, AUTHORS.txt (
 * Dual licensed under the MIT or GPL Version 2 licenses.
 * Depends:
 * jquery.ui.core.js
 * jquery.ui.widget.js
 *  jquery.ui.button.js
 * jquery.ui.draggable.js
 * jquery.ui.mouse.js
 * jquery.ui.position.js
 * jquery.ui.resizable.js

Now we can see what dialog depends, dialog also depends jquery.ui.dialog.css. Let's change it for now and repeat this for other module accordion, autocomplete, button, datepicker, dialog, progressbar, resizable, selectable, slider, tabs, for example, add jquery.ui.accordion.css in jQuery.ui.accordion.min.js header. We will need it later in using next method.

* Depends:
 * jquery.ui.dialog.css
 * jquery.ui.core.js
 * jquery.ui.widget.js
 *  jquery.ui.button.js
 * jquery.ui.draggable.js
 * jquery.ui.mouse.js
 * jquery.ui.position.js
 * jquery.ui.resizable.js


To convert this into code, we can write the following. Please note that I am not using the url of resource as the resource key, later I will map the reource key to url.

matrix.depend( "ui.dialog.js": "ui.dialog.css, ui.core.js, ui.widget.js, ui.button.js, ui.draggable.js, 
ui.mouse.js,ui.position.js, ui.resizable.js");

But this is not enough, because inner resources have their own dependencies as well. So we need to open all direct and indirect referenced js files and find out their dependencies. If there a file has no dependencies, we still need to use a null value. For example, ui.core.js does not have any dependencies, so its dependencies is null. Finally, we can write something like the following.

matrix.depend( {
 "ui.dialog.js": "ui.dialog.css, ui.core.js, ui.widget.js, ui.button.js, ui.draggable.js, ui.mouse.js, 
ui.position.js, ui.resizable.js",
 "ui.button.js": "ui.button.css, ui.core.js, ui.widget.js",
 "ui.draggable.js": "ui.core.js, ui.mouse.js, ui.widget.js",
 "ui.mouse.js": "ui.widget.js",
 "ui.resizable.js":"ui.resizable.css, ui.core.js, ui.mouse.js, ui.widget.js",
 //ui.redmond.css is my favorite skin for jQuery ui, :)
 "dialog.module" : "ui.core.css, ui.redmond.css, ui.dialog.js"
} );

Now we need to solve the second problem, resolving the url of resource, and we can use the following

matrix.url( {
 "ui.core.css": "js/jquery.ui/css/base/jquery.ui.core.css",
 "ui.dialog.css": "js/jquery.ui/css/base/jquery.ui.dialog.css",
 "ui.resizable.css": "js/jquery.ui/css/base/jquery.ui.resizable.css",
 "ui.button.css": "js/jquery.ui/css/base/jquery.ui.button.css",
 "ui.smoothness.css": "js/jquery.ui/css/smoothness/jquery.ui.theme.css",
 "ui.redmond.css": "js/jquery.ui/css/redmond/jquery.ui.theme.css",
} );

matrix( "dialog.module", function () {
 $( "#linkDialog" ).click( function() {
  $( "<h1>hello</h1>" ).dialog();
  return false;
 } );
} );

Let's how the resources are actually process.

So far so good, the fetching is parallel, and the evaluation is serial and in the correct order of dependencies. But the above configuration is too tedious. But if we want to use accordion widget, we need to repeat this process, obviously it violate the principle "Don't Repeat Yourself". Let's do "Convention over configuration". As I mentioned before, resource is extensible, you can extend handling of resource by adding a new resource type, which map to a new handler. For example, and x.mytype is mapped to a handler matrix.handlers.mytype. Here is methods that a handler can implement.

  1. load(resourceKey) [must implement]

    This method is to fetch and evaluate the resource and its dependencies into memory.

  2. release(resourceKey) [optional]

    This method is to release single resource (not its dependencies)

  3. url(resourceKey) [optional]

    This method is to build a url using a naming convention

  4. parseDepends(sourceCode) [optional]

    how to parse source code to build dependencies

Only the load method need to be implemented, other methods are optional. In the case here however, we want to reuse the default load method of javascript handler and css handler, all we need to do use extend url, and parseDepends. The following shows how to use the addHandler method

// it return back the new handler
//newHandlerName is also the name of new resource type
//matrix.addHandler(newHandlerName, baseHandlerName [,newBehavior])

matrix.addHandler( "uicss", "css", {
 url: function ( resourceKey ) {
  return matrix.fullUrl( matrix.baseUrl + "jquery.ui/css/base/jquery.ui." + 
matrix.resourceName( resourceKey ) + ".css" );
} );

matrix.addHandler( "uitheme", "css", {
 url: function ( resourceKey ) {
  return matrix.fullUrl( matrix.baseUrl +  "jquery.ui/css/" + 
matrix.resourceName( resourceKey ) + "/jquery.ui.theme.css" );
} );

var rdependencies = /^\s*\/\*[\w\W]*Depends:([\w\W]+)\*\//,
 ruiModule = /jquery\.ui\.(\w+?)\.(js|css)/gi;

//extend js handler
matrix.addHandler( "uijs", "js", {

 url: function ( resourceKey ) {
  return matrix.fullUrl( matrix.baseUrl +  "jquery.ui/jquery.ui." + 
matrix.resourceName( resourceKey ) + ".min.js" );

 parseDepends: function ( sourceCode ) {
  var dependencies = [],
   dependText = rdependencies.exec( sourceCode );

  if ( dependText = dependText && dependText[1] ) {
   //dependText is something like
   * jquery.ui.tabs.css
   * jquery.ui.core.js
   * jquery.ui.widget.js
   while ( module = ruiModule.exec( dependText ) ) {
    //find and convert it to xxx.uicss
    //find and convert it to xxx.uijs
    // "xx" + ".ui" + "js" == "core.uijs"
    // "xx" + ".ui" + "css" == "core.uicss"
    dependencies.push( module[1] + ".ui" + module[2] );
   return dependencies.length ? dependencies.toString() : null;

  return null;
} );

In the above code, the url method is straight forward, it is used to map a resource key to a url, since jQuery UI team follow a naming convention of their components, we can use this, and use a function to calculate the url based on resource key. What is interesting this the parseDepends method. The idea is that if we can parse it by human eye, we can parse it by code. The method basically extract the headers we modified before, and convert them into resource keys, and then build dependencies. With all this code in place, we can throw away the previous dependencies and url configuration. Let's write a bit of code, to register the dependencies and the locations of all jQuery ui files.

matrix.baseUrl = "js/";

matrix.depend( {
 "uicss.module": "core.uicss, redmond.uitheme"
} );

("accordion,autocomplete,button,datepicker,dialog," + 
"progressbar,resizable,selectable,slider,tabs," + 
"draggable,droppable,mouse,position,sortable").split( "," ),
 function (index, value) {
  var resourceKey = value + ".module";
  var depends = value + ".uijs, uicss.module";
  matrix.depend(resourceKey, depends);
 } );

I agree that I need to spend more time to do this kind of convention over configuration. But I think it is worthy, because I can dynamically load the the resources of just of my widget, any widget in jQuery UI, not just dialog. In the next post, I will explain dependencies refresh, release release, performance and other advanced topic in using matrix.

Apr 3, 2011

tweaking HTTP header to imporve performance

Lately, I am doing some client side optimization. One of the techniques is cache. When browser firstly request a resource and get the response, the cache subsystem of the browser need to determine whether to cache resource. It is based on the response header Cache-Control, and Expires. When neither of them exists in the response header, browsers use their own strategy to cache, these strategies vary from browser to browser, so they are not predictable, we should explicitly set these headers. To explicitly disable cache in browser, we can use the following

Pragma: no-cache 
Cache-Control: no-cache 
Cache-Control: no-store

The Pragma: no-cache header is included in HTTP 1.1 for backward compatibility with HTTP 1.0+. It is technically valid and defined only for HTTP requests; however, it is widely used as an extension header for both HTTP 1.0 and 1.1 requests and responses. HTTP 1.1 applications should use Cache-Control: no-cache, except when dealing with HTTP 1.0 applications, which understand only Pragma: no-cache. According RFC 2616, Cache-Control: no-cahce is semantically different from Cache-Control: no-store. non-cache still allows browser cache a response, and the cache needs to re-validate the response with the origin server before serving it. no-store request browser not to cache response at all. However most browser treat "no-cache" as "no-store".
To implement the semantics of "no-cache", we should use Cache-Control: max-age=0 or Cache-Control: s-maxage=0 or Cache-Control: must-revalidate.

To explicitly to cache resource, we should use the following.

Cache-Control: max-age=3600
Cache-Control: s-maxage=3600
Expires: Fri, 05 Jul 2002, 05:00:00 GMT

The Cache-Control: max-age header indicates the number of seconds since it came from the server for which a document can be considered fresh. There is also an s-maxage header (note the absence of a hyphen in "maxage") that acts like max-age but applies only to shared (public) caches. The deprecated Expires header specifies an actual expiration date instead of a time in seconds. The HTTP designers later decided that, because many servers have unsynchronized or incorrect clocks, it would be better to represent expiration in elapsed seconds, rather than absolute time. An analogous freshness lifetime can be calculated by computing the number of seconds difference between the expires value and the date value.

If browser hit the same url again, browser will firstly ask the cache sub-system to get a cache copy. If the cache is still fresh based on "Cache-Control" or "Expires" header, the cache is returned, and no request is sent to server. If the cache expired, determined whether a validator was send from the previous response, different request will be sent. If a validator of previous response is "Last-Modified", the following will apply.

//validator of previous response
Last-Modified: Sun, 03 Apr 2011 14:34:43 GMT

//validator of next request
If-Modified-Since: Sun, 03 Apr 2011 14:34:43 GMT

If a validator of previous response is "Etag", the following will apply

//validator of previous response
ETag: 100

//validator of next request
If-None-Match: 100

Normally we should use either "Last-Modified" or "ETag" in response from server. If neither of them is available in the previous response, then a new request will make. You should note that validator is independent with "Cache-Control" and "Expires" header, which means it can be used without these two header.

In, if you don't do anything, the following default header will be generated.

HTTP/1.1 200 OK
Server: ASP.NET Development Server/
Date: Sun, 03 Apr 2011 14:47:30 GMT
X-AspNet-Version: 4.0.30319
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 509
Connection: Close

Because the Cache-Control header does not specify a max-age, by default is zero, and because there is no validator available, is semantically "Cache-Control: no-store". So we don't need to do anything if we want the client not to cache the response, because it is by default. The following is the default header of next response.

GET http://localhost:1491/WebSite1/Cache.aspx HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Accept-Language: en-US
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Host: localhost:1491
Cookie: ASP.NET_SessionId=imdo5jrzxkpw2ohy24guaw0b

The following is a typical header to enable client side cache.

Cache-Control: max-age=3600
Last-Modified: Sun, 03 Apr 2011 14:34:43 GMT

Cache-Control: max-age=3600
ETag: 100

We should prefer "Cache-Control" over "Expires" because it is smarter. When the resource is static, we should config web server to add a Last-Modified header but no using ETag, because it simply and easier. And ETag is problematic in IIS web farm. When the resource is dynamic, we should use dynamically generate an ETag as hash to compare change. Here is some server side code.

private void AddCacheControlHeader(int seconds)
        Response.AddHeader("Cache-Control", string.Format("max-age={0}", seconds));

    private void AddETagValidator()
        string hash = "xyz"; //calcuate a hash match the request
        Response.AddHeader("ETag", hash);

    private void ProgessETagValidator()
        string hash = "xyz"; //calcuate a hash match the request
        if (Request.Headers["If-None-Match"] == hash)
            Response.StatusCode = 304;

Mar 19, 2011

A enhanced curry method

JavaScript is a functional language, it supports function composition, we can do curry with JavaScript. Douglas Crockford has defined "curry" method in his book JavaScript: The Good Parts to facilitate the steps to define curried function.

Function.prototype.method = function ( name, func ) {
 if ( !this.prototype[name] ) {
  this.prototype[name] = func;

Function.method( 'curry', function () {
 var slice = Array.prototype.slice,
   args = slice.apply( arguments ),
   that = this;
 return function () {
  return that.apply( null, args.concat( slice.apply( arguments ) ) );
} );

//so I can write the following code

test( "test curry", function () {
 function cancat_abc( a, b, c ) {
  return a + b + c ;

 var cancatBc = cancatAbc.curry( "a" );
 var result = cancatBc( "b", "c" );
 equal( result, "abc", "the first parameter is preloaded" );

} );

Nice. But there is problem here, the preloaded arguments have to be the first consecutive arguments. So if I have want to preload the second and third parameter. The curry method does not support that.

Of course, I can do it manually instead of using the curry method, but this is not quite DRY (Don't Repeat Yourself). So I modify the curry method as follow to do the work.

Function._ = {};

Function.method( 'curry', function () {
 var slice = Array.prototype.slice,
   preloads = slice.apply( arguments ),
   _ = Function._,
   fn = this;
 return function () {
  var args = slice.apply( arguments );
  for ( var i = 0; i < preloads.length; i++ ) {
   if ( preloads[i] == _ ) {
    preloads[i] = args.shift();

  return fn.apply( null, preloads.concat( args ) );
} );

var _ = Function._;

test( "test curry", function () {
 ok( _, "an dummy object has been defined" );
 function cancat_abcd( a, b, c, d ) {
  return a + b + c + d;

 var cancatBd = cancatAbc.curry( "a", _, "c", _ );
 var result = cancatBd( "b", "d" );
 equal( result, "abcd", "curry should work in order" );

} );

the when method

In jQuery 1.5, there is a "when" method.

function when(objects) {}

//we can call with like
jQuery.when(object1, object2, object3, ..).then(done, fail);

The semantics of the function is when ALL the objects'(object1, object2, ..) value is realized, then call the done function with all the realized value, when one of them fail, call the fail function. These objects can be a "realized" value or a promise (promising value). If all the value are already "realized" initially, the done function will be called synchronously.

the promise object

The jQuery.Deferred object has a function .promise(), it returns a promise object. Some other people call it future. In the async framework of .net, it is called Task<T> . It an object that promise to deliver a value in the future. So if believe that promise, you can express your believing like this

promise.done(function () {
  var valuesOfFuture = [], 0);

The promise object does not has method of "resolve" and "reject". Because these method belong to server side. This pattern is very popular, but jQuery stream line this pattern. It is very useful pattern.

jQuery.Deferred Object

Internally, the Deferred object is implemented with two _deferred object. The Deferred object offer two set of functions, done/resolveWtih/resolve/isResolved and fail/rejectWith/reject/isRejected. The are basically the functions of two internal _deferred object. The failed set function is the done/resolveWith/resoved/isREsolved function of fail object. But we don't have cancel function, because it is unnecessary, it is replaced with reject function. The done function will empty the failed list, and the reject function will empty the done list.

It also introduced two new functions, then and promise. The then function is short cut to push a done function and failed function at one shot. We care about the success and compensation at the same time, we should consistently use then function instead of interweaving done or fail method. But if we don't care about that, we can interweave done and fail method. However, I think the then method should always be used, as it adds very little cost.

Another new method is promise method. It basically return a new object or decorate an existing object with "promise prospect" or promise methods then, done, fail, isResolved, isRejected, promise. The promise object does the same thing as Deferred object, except it can not resolve, reject, which should be used at the server side. When we first call the d.promise(obj) with a real obj, then the obj is augmented with the promise methods and cached internally, and this pattern should be used at the first time. The subsequent call d.promise() will return the cached object.

The constructor Deferred(initialize) can be passed with an initialize function. Your function can be defined as

function (deferred) {
  //this == deferred
   this.then(x, y);
   this.m1 = "m1";

jQuery._Deferred Object

The jQuery._Deferred is an object maintaining a task list. To understanding it, help you to understand how the jQuery.Deferred and jQuery.when works. The _Deferred object has the following use case.

  • Push your task into the list to be run(resolved) in the future. It has been resolved, call the resolved method again.

  • Resolve(run) the tasks in the list, the task will be run in the order you push it, [].shift method is used to dequeue from the list. If everything runs fine, the task will be empty.

  • Push the task into task list during the task is run. But the new task will the last one to run.

  • Exception handling inside of your task. If the current task is run synchronously, you can optionally use exception handling, an exception is not caught, the remaining tasks will be canceled.

  • You have an option to eagerly cancel the remaining task, in the current running task. This is an advanced task.

  • Once you resolve the task list one a set of values, then the resolve context, and values are cached. If you resolve it again with new values, the new task still be run, but run in the old context, and values.