Asynchronous Programming in Server

To clarify, in this article I discuss about Asynchronous Programming (AP) rather than Asynchronous Request and they are different. By definition, Asynchronous request is a request that lets the client do other work while the request is being processed, enhancing parallelism within an application. Asynchronous Programming refer to a method of programming where you let the work executing in another thread and let the main thread free to do other work. As Asynchronous Programming is getting more popular these days I want to elaborate on the need of it and recent improvement in Java that make Asynchronous Programming easier to use.

WHY SHOULD WE USE AP

Most of use has been used to Async Programming from the beginning day of Java when dealing with UI. Swing use one thread to draw the graphic and event handling. Hence, when the system is busy processing the event, the UI virtually hang. To avoid this, we were told to keep the handling event code as simple as possible. If we feel that the work will take long time to run, we must let it executed in another thread.

However, as Java Swing become less and less popular nowadays, most developers do not need to deal with asynchronous programming that often any more and the word "asynchronous" seem more related to web protocol.

Still, asynchronous programming is not only applicable to UI. Lots of logic on server side benefit tremendously from proper use of AP. Scheduling, Input/Output processing and background processing all requires developers to process work in a separate thread to avoid blocking of main thread or long waiting.

With the trend of hardware developers introducing multiple cores CPU and high bandwidth network, Asynchronous becoming more and more important in programming.

HOW SHOULD WE USE AP
Let's cover three most popular scenarios of Asynchronous Programming

1/ Fire and forget

The simplest form of Asynchronous Programming is to create a runnable object that contain the business logic and create a thread to run it. Your main thread can happily terminate after this.  For example, client send a request to server to start a background process. The triggering of background process complete and the response is sent back to client.

2/ Collaborate of work

In the second scenario, you need to spawn multiple threads to complete a heavy task but your main thread cannot exit without the task completed. For real life example, we can think of spamming emails to all of your clients. The work is huge but separable and can be done concurrently. The simplest way to do things is to create one array of runnable objects and create one thread to execute each of them. However, this simple method is strongly un-recommended as we may not know the amount of runnable we need to create. Therefore, we may create unknown amount of threads, which may throw exceptions (whether due to out of memory or limit file handler, depend on which threshold reach first). The better solution in this scenario should be usage of thread pool executor and submit all your runnable into array blocking queue. With this strategy, executor will only use a fixed amount of thread to execute your code and the thread can be reused to execute other tasks.

Recently, Java introduce a fork and join framework. It suppose to out-perform Executor method above with Divide and Conquer problems. I will cover it in another article.

3/ Scheduling

The simplest way for Scheduling is to start-up a thread, do some work and sleep for a while, then repeat the process again. I would not recommend this as well because Executor seem to be a better option. Even a single thread executor still give you more power as you can schedule with fixed interval rather than fixed delay (sleep actually provide you a fixed delay rather than fixed interval of execution). If you use some of the scheduling framework like Quartz, it is even more recommended as you have a declarative style of scheduling rather than coding.

In a similar scenario, when you need to schedule a work in the future but not repetitive, executor still a better choice. In this case, you create a FutureTask object that wrap around the piece of code that you need to execute in the future and submit it to the executor. This work well and clean. Spawning a thread to wait for the time to come to execute the task is unforgivable.

HOW TO WRITE BETTER AP CODE

1/ Know how to deal with multi-thread issues

To write better AP code you should understand the common issues with Asynchronous. Asynchronous is effectively multi-thread; hence, suffer from all multi-thread issues like deadlock and thread safe. You should pay attention to your runnable object. There is no point to create multiple thread to do the work concurrently if your runnable object have a long synchronize block and even can cause deadlock.

2/ Control the amount of threads and reuse as much as possible

Thread is expensive to create. Hence, please use it responsibly. It is highly recommended that you always know exactly how many threads is created by you are running at any moment. Using thread pool allow you to limit the amount of thread and reuse the object to execute new task.

3/ Avoid sleeping/unproductive time

Not all the threads are active all the time. It is quite often that a thread wake up, check for a condition to execute and if the condition not exist, go back to idle. In this case, there may be wasting time if the condition is ready but your thread is not executed. The basic guideline is to try to minimize this wasting time. There are two methods to detect if a condition is matched, polling and callback. With polling, you constantly check with the fixed interval to see if the condition is matched. With callback, you have the luxury of waiting until someone wake you up to do the work.

For most of the time callback give you less delay. Therefore I would advise you to use callback if possible. Still, kindly remember that callback may not guaranteed that you will be notified immediately if there are more than one listener register them selves to the event. Some of the event handler framework that I have seen sequentially execute listener codes in a single thread. Because of this, you may need to wait for some other listeners to complete before being notified. In worst case, your callback mechanism may perform worse than polling.

RECENT IMPROVEMENT

As you may already know, thread have it own stack memory to store local variable, method and return value. As a direct consequence, thread when executing a method will not be available for any other task, even if you put it to sleep. Unless this method is completed, it is likely that your thread is not available to reuse.

In the earlier day, most of server implement thread per connection for HTTP 1.1. Needless to say, this approach put heavy burden for server and inefficient, especially when the user only make a request in a while. Today, most of popular web servers serving thread per request rather than per connection. The threads are provided by HTTP thread pool, which have a upper limit on the amount of the HTTP threads it can issue. The Http Thread is blocked from the time the request constructed and pass to HTTP Servlet until the servlet return a response. With the knowledge of how thread allocate memory above, this seem to be the limit until recent breakthrough in Java API.

The problem to solve is the blocking of HTTP thread when waiting for a long process to run until getting a response. Even if we try to put the long running process to a separate thread, the http thread is still blocked, waiting to render the response; hence, no benefit can be seen. Because of this, changes have been made to make the HTTP thread reusable when the long running work is happening on the background. On this matter, I want to discuss two strategies that widely adopted in the market as in Servlet API 3.0 and Play framework.

Servlet API 3.0 is a set of API. Therefore, it attempt to fix the problem with introducing of new API. Oracle provide this example to illustrate the usage of AsyncContext

 AsyncContext aCtx = request.startAsync(req, res);
 ScheduledThreadPoolExecutor executor = new ThreadPoolExecutor(10);
 executor.execute(new AsyncWebService(aCtx));

The doGet/doPost method can return immediately after constructing the AsyncContext. However, a response has not been generated here. When the Asynchronous task is completed, it render the response and the web container finally response to user request. I think this API is clean but it impose a limit on sharing code between the original doGet method and the asynchronous task. If there is anything you want to share, it must be passed as parameter to AsyncWebService constructor.

Play framework follow a different approach. As the byte code executed is enhanced by Play framework, it do not bother to do much on introducing new API to solve the problem. Here is a sample I create using Play framework

public class Application extends Controller {

    public static void index() {
     String welcome = "synchronous text";
     Promise<String> welcomeJob = new Job<String>(){
      public String doJobWithResult(){
       try {
     Thread.sleep(5000);
    } catch (InterruptedException e) {
     e.printStackTrace();
    }
       return "async text";
      }
     }.now();
     String awaitResult = await(welcomeJob);
        renderText(welcome + awaitResult);
    }
}

Look at the code above, you only see one method index() that render the response. However, what really happen after byte code enhancement if the split of the method to 2 methods, before and after await(). The two methods are perfectly available to be executed by different HTTP Thread. It is interesting that Play manage to restore the thread state perfectly and I can use the local variable "welcome" that I create on the other thread. This is quite amazing.