HTTP Request processing in details

From Morfik Wiki

The main role of the server-side code of a Morfik XApp is to receive and respond to Http requests. These Http requests usually come from the users of your XApp via a web browser, although they can come from other programs. When an XApp receives an Http request, it may need to perform certain actions with the data received (e.g. save a record to the database), or return certain data in the Http response (e.g. the html code of a form).

An Overview of HTTP Request Processing

The Http request processing operation is the “be all and end all” of all activities that take place in the server side within a Morfik application. The diagram in Figure 1 shows the key components involved in this operation.

Figure 1 - Http Request Processing

Upon arrival of a request, the Web server creates a new execution thread and invokes the main entry point inside the Morfik application framework. The application's main object (a descendant of TXApp, eg. Project1XApp) receives the call and immediately creates a pair of Request and Response objects. By parsing the URL the application's main object determines the class of the HTTP server that is to serve the request and compose the response.

Before creating an instance of the HTTP server, all HTTP Filters that are installed into the framework are executed. The role of HTTP Filters are to alter or block an incoming request before the request is processed. Once a rquest gets through HTTP filters, an instance of the HTTP server is created and its execute method is invoked.

The execute method performs the main task in http request processing operation. Depending on the type of the HTTP server, the execute method performs tasks that vary from generating HTML and PDF to executing the code within a WebMethod.

Note: One important thing to bear in mind is that there could be multiple request processing operations in progress at any point in time within the server side of the framework. Each one of these operations run on its own execution thread and as such the execution of the server side code is highly concurrent. This includes all the application code written in the server side modules. Thread safety is an important issue on the server side and care must be taken in order to ensure threads are properly synchronized when accessing shared data. A typical example of this is when writing into an external file. The code that writes into the file must be written in such a way that only allows one thread of execution to run through it at any given point in time.

HTTP Request

An Http request is sent to a specific server, identified either by a host name (e.g. www.morfik.com) or by an IP address (e.g. 67.192.129.155). It includes the following:

The Morfik framework provides the THttpRequest class for working with the Http request. This can be accessed via the Request member of the THttpServer class. For example, the following code shows how one could log the IP address from which a webmethod call originated:

FX Code

Procedure WebMethod1.Execute;
Begin
    WriteToXAppLog('RemoteAddress=' + SoapServer.Request.RemoteAddress);
End;

Public Overrides Sub Execute
    WriteToXAppLog("RemoteAddress=" & SoapServer.Request.RemoteAddress)
End Sub

public override void Execute()
{
    WriteToXAppLog("RemoteAddress=" + SoapServer.Request.RemoteAddress);
}

Here is an example of an Http request:

POST /Index.htm?sbd=2&lng=&thm=ProjectTheme&tzo=-660&ins=Index&xid=%7B7166F464-9484-4DE3-8F44-
                   F00CF7C9C74B%7D&sid=&cls=form&ViewMode=vmSingle&PageName=Home HTTP/1.1
Accept: */* 
Accept-Language: de-at 
Referer: http://www.morfik.com/
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB6.3; SLCC1; 
                         .NET CLR 2.0.50727; Media Center PC    5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729)
Host: www.morfik.com 
Content-Length: 0 
Connection: Keep-Alive
Pragma: no-cache
Cookie: __utma=57227574.1756966090.1260398203.1260491327.1260491779.5;
          __utmb=57227574;__utmz=57227574.1260398203.1.1.utmccn=(direct)|
            utmcsr=(direct)|utmcmd=(none); __utmc=57227574 

[message body]

HTTP Response

In response to an Http request, the server returns an Http response. This includes the following:

The Morfik framework provides the THttpResponse class for working with the Http response. This can be accessed via the Response member of the THttpServer class. For example, the following code shows how one could force a form's html code to be returned in uncompressed format:

FX Code

Procedure Index.WebFormAfterExecute(Sender: TWebDocument);
Var
    HttpResponse : THttpResponse;
Begin
    HttpResponse := HttpServer.Response;
    HttpResponse.BlockCompression := True;
End;

Published Message Sub WebFormAfterExecute(Sender As TWebDocument)
    Dim HttpResponse As THttpResponse
    HttpResponse = HttpServer.Response
    HttpResponse.BlockCompression = True
End Sub

published message void WebFormAfterExecute(TWebDocument Sender)
{
    THttpResponse HttpResponse;
    HttpResponse = HttpServer.Response;
    HttpResponse.BlockCompression = True;
}

Example of an Http response:

HTTP/1.1 200 OK 
Cache-Control: no-store
Allow: Post
Content-Type: text/html; charset=utf-8
Content-Encoding: deflate
Last-Modified: Fri, 11 Dec 2009 00:51:37 GMT
Server: Microsoft-IIS/7.0
Content: 
Date: Fri, 11 Dec 2009 00:51:37 GMT
Content-Length: 2807

[message body]

HTTP Servers

Pivotal to HTTP request processing are classes that descend from THttpServer. The THttpServer class is an abstract class that represents the processing of an Http request in the server-side code of an XApp. THttpServer has two memebers, Request and Response that provide access to the underlying Http request and response details. The key method of THttpServer class is the Execute method. The Execute method is overridden in descendant classes to perform the required handling of the request data, and to compose the response.

These classes provide the high level context for processing incoming requests. The application high level objects (Forms, Reprt, WebMethods, etc) have a member field that provides access to the underlying Http Server object as shown in the code below:

FX Code

Procedure About.TextLabel3BeforePrint(Sender: TWebControl; Canvas: TWebCanvas; Var Print: Boolean);
Begin
    TextLabel3.Caption := HTTPServer.Request.SecurityID;
End;

Published Message Sub TextLabel3BeforePrint(Sender As TWebControl, Canvas As TWebCanvas, ByRef Print As Boolean)
    TextLabel3.Caption = HttpServer.Request.SecurityID()
End Sub

published message void TextLabel3BeforePrint(TWebControl Sender, TWebCanvas Canvas, ref Boolean Print)
{
    TextLabel3.Caption = HttpServer.Request.SecurityID();
}

Note: In the case of WebMethods the corresponding field that provides access to the underlying HTTP server object is is SoapServer.

The Morfik framework provides the following THttpServer descendant classes for handling different kinds of Http requests.

The built in Http servers are involved with key operations inside a Morfik application. They rely on compiler to generate application-specific code for objects that they manipulate. Their behavior is generally fixed, although can be overridden through a descendant class and subsequent registration of the server class as a custom server.

Custom HTTP Servers

Custom Http servers can be used to extend the capabilities of the framework beyond what is provided by the default built-in Http servers. Custom http servers are descendants of THTTPServer and can be defined inside any of the application server side modules.

A custom Http server must be made known to the framework so it can be called upon to perform its function. There are two ways to make a custom Http server known to the framework.

1- Registering a custom Http Server to serve a specific type of Http requests

The following function in SystemServer module registers a custom Http server class with the framework:

Procedure RegisterCustomHttpServer(Const RequestCls : String; ServerClass : THttpServerClass);

The first parameter specifies the type of Http request for which this server is to be invoked.

The following sample code shows a custom Http server that returns server's local time:

FX Code

Unit Module1;
 
Interface
 
Uses
   SystemServer,
   SystemUtilities;
 
Type
TPingServer = Class(THttpServer)
   Procedure Execute; Override;
End;
 
Implementation
 
Procedure TPingServer.Execute;
Begin
    Response.WritelnString(Now().toLongDateString);
End;
 
Initialization
   RegisterCustomHttpServer('ping', TPingServer);
End.

Imports SystemServer
Imports SystemUtilities
 
Namespace Module1
    Public Class TPingServer
        Inherits THttpServer
        Public Overrides Sub Execute
            Response.WritelnString(Now().ToLongDateString())
        End Sub 
 
    End Class
    Initialization
        RegisterCustomHttpServer("ping", TPingServer)
    End Initialization
End Namespace

imports SystemServer;
imports SystemUtilities;
 
namespace Module1
{
    public class TPingServer : THttpServer
    {
        public override void Execute()
        {
            Response.WritelnString(Now().ToLongDateString());
        }
    }
    initialization
    {
        RegisterCustomHttpServer("ping", TPingServer);
    }
}

2- Overriding the GetHttpServer method of TXApp class

By overriding the TXApp.GetHttpServer method one can have a greater level of control over the way request checking is performed. If the request is checked and found to be targeting the custom Http server, an instance of the custom server is created and returned as the result of the TXApp.GetHttpServer method. The following sample code shows a custom Http server that returns server's local time:

FX Code

Unit Project1XApp;
 
Interface
 
Type
Project1XApp= Class(TXApp)
  Published
    { Events }
  Private
    { Private declarations }
  Public
    { Public declarations }
    Function GetHttpServer(Request : THttpRequest; Response : THttpResponse): THttpServer; Override;
End;
 
Implementation
 
Type
TPingServer = Class(THttpServer)
   Procedure Execute; Override;
End;
 
Procedure TPingServer.Execute;
Begin
    Response.WritelnString(Now().toLongDateString);
End;
 
Function Project1XApp.GetHttpServer(Request : THttpRequest; Response : THttpResponse): THttpServer;
Begin
    If Request.GetURLParameterValue('cls').Equals('ping') Then
        Result := TPingServer.Create(Request, Response)
    Else
        Result := Inherited GetHttpServer(Request,Response);
End;
 
End.

Namespace Project1XApp
    Public Class Project1XApp
        Inherits TXApp
        Public Overrides Function GetHttpServer(Request As THttpRequest, Response As THttpResponse) As THttpServer
            If Request.GetURLParameterValue("cls").Equals("ping") Then
                Result = New TPingServer(Request, Response)
            Else
                Result = MyBase.GetHttpServer(Request, Response)
            End If
        End Function 
 
    End Class
 
    Private Class TPingServer
        Inherits THttpServer
        Public Overrides Sub Execute
            Response.WritelnString(Now().ToLongDateString())
        End Sub 
 
    End Class
 
End Namespace

namespace Project1XApp
{
    public class Project1XApp : TXApp
    {
        public override THttpServer GetHttpServer(THttpRequest Request, THttpResponse Response)
        {
            if (Request.GetURLParameterValue("cls").Equals("ping"))
            {
                Result = new TPingServer(Request, Response);
            }
            else
            {
                Result = base.GetHttpServer(Request, Response);
            }
        }
    }
 
    private class TPingServer : THttpServer
    {
        public override void Execute()
        {
            Response.WritelnString(Now().ToLongDateString());
        }
    }
 
}

HTTP Filters

Sometimes you may wish to block specific Http requests received by your XApp. For example, you may wish only to accept requests coming from certain IP addresses, or to limit the size of uploaded files. Such tasks may be accomplished by means of Http filters.

An Http filter consists of a function, which must conform to the following signature:

THTTPFilterProc = Function (HTTPServer : THTTPServer; UserData : Pointer) : Boolean;

The Http filter function should return false and/or raise an exception if the request is to be blocked; otherwise it should return true. It needs to be registered by calling the following procedure, declared in SystemServer.mmd:

Procedure RegisterHttpFilter(FilterProc : THTTPFilterProc; UserData : Pointer);

where FilterProc is a pointer to the Http filter function. UserData is a pointer to user-defined data (may be left as nil).

Example 1. By default, the Morfik framework rejects all Http requests larger than a certain limit. The limit is 5 megabytes for file upload requests and 1 megabyte for all other requests. These limits are enforced by means of the following Http filter, declared in SystemServer.mmd:

FX Code

Function CheckAllowPost(HTTPServer : THTTPServer; UserData : Pointer) : Boolean;
Const
    HTTP_ERROR_413 = 'Http Error 413 - Request too large';
Begin
    Result := True;
    If StringsEqual(HTTPServer.Request.RequestMethod,'POST') Then
       If Not Catalog.postLimits.canAccept(HTTPServer.Request.GetURLParameterValue('cls'),
                                           StrToIntDef(HTTPServer.Request.ContentLength, -1)) Then
    Begin
        StrCopy(HTTPServer.Request.ServerImplementation.LogData, HTTP_ERROR_413);
        Raise Exception.Create(HTTP_ERROR_413);
    End;
End;
 
...
 
 RegisterHttpFilter(CheckAllowPost, Nil);

Private Function CheckAllowPost(HTTPServer As THttpServer, UserData As Pointer) As Boolean
    Const HTTP_ERROR_413 = "Http Error 413 - Request too large"
    Result = True
    If StringsEqual(HTTPServer.Request.RequestMethod, "POST") Then
        If Not Catalog().PostLimits.CanAccept(HTTPServer.Request.GetURLParameterValue("cls"), 
                                               StrToIntDef(HTTPServer.Request.ContentLength, -1)) Then
            StrCopy(HTTPServer.Request.ServerImplementation.LogData, HTTP_ERROR_413)
            Throw New Exception(HTTP_ERROR_413)
        End If
    End If
End Function
 
...
    RegisterHttpFilter(CheckAllowPost, Nothing)

private Boolean CheckAllowPost(THttpServer HTTPServer, Pointer UserData)
{
    const HTTP_ERROR_413 = "Http Error 413 - Request too large";
    Result = True;
    if (StringsEqual(HTTPServer.Request.RequestMethod, "POST"))
    {
        if (!Catalog().PostLimits.CanAccept(HTTPServer.Request.GetURLParameterValue("cls"), 
                                           StrToIntDef(HTTPServer.Request.ContentLength, -1)))
        {
            StrCopy(HTTPServer.Request.ServerImplementation.LogData, HTTP_ERROR_413);
            throw new Exception(HTTP_ERROR_413);
        }
    }
}
 
...
    RegisterHttpFilter(CheckAllowPost, null);

Example 2. We have a top-secret report, that should not be accessible to anybody outside the organisation. As a security measure, we add an Http filter to our XApp, that blocks any request for this report that does not come from a whitelist of known IP addresses.

We register our Http filter along with the list of IP addresses as follows:

FX Code

Procedure Project1XApp.XAppStart(Sender: TObject);
Var
    AllowedIPAddresses : TStringList;
Begin
    AllowedIPAddresses := TStringList.Create;
    AllowedIPAddresses.Sorted := True;
    AllowedIPAddresses.Add('67.192.129.155');
    AllowedIPAddresses.Add('67.15.211.4');
    RegisterHttpFilter(HttpFilter2, AllowedIPAddresses);
End;

Published Message Sub XAppStart(Sender As TObject)
    Dim AllowedIPAddresses As TStringList
    AllowedIPAddresses = New TStringList()
    AllowedIPAddresses.Sorted = True
    AllowedIPAddresses.Add("67.192.129.155")
    AllowedIPAddresses.Add("67.15.211.4")
    RegisterHttpFilter(HttpFilter2, AllowedIPAddresses)
End Sub

published message void XAppStart(TObject Sender)
{
    TStringList AllowedIPAddresses;
    AllowedIPAddresses = new TStringList();
    AllowedIPAddresses.Sorted = True;
    AllowedIPAddresses.Add("67.192.129.155");
    AllowedIPAddresses.Add("67.15.211.4");
    RegisterHttpFilter(HttpFilter2, AllowedIPAddresses);
}

The Http filter function looks like this:

FX Code

Function HttpFilter2(HTTPServer : THTTPServer; UserData : Pointer) : Boolean;
Var
    Temp : Integer;
Begin
    If HttpServer.Request.URLDocumentName.Equals('Top Secret Report') Then
        Result := TStringList(UserData).Find(HttpServer.Request.RemoteAddress, Temp)
    Else
        Result := True;
End;

Private Function HttpFilter2(HTTPServer As THttpServer, UserData As Pointer) As Boolean
    Dim Temp As Integer
    If HTTPServer.Request.UrlDocumentName.Equals("Top Secret Report") Then
        Result = Ctype(UserData, TStringList).Find(HTTPServer.Request.RemoteAddress, Temp)
    Else
        Result = True
    End If
End Function

private Boolean HttpFilter2(THttpServer HTTPServer, Pointer UserData)
{
    Integer Temp;
    if (HTTPServer.Request.UrlDocumentName.Equals("Top Secret Report"))
    {
        Result = (TStringList)(UserData).Find(HTTPServer.Request.RemoteAddress, Temp);
    }
    else
    {
        Result = True;
    }
}

This blocks any request for „Top Secret Report“ coming from a non-whitelisted IP address, but allows all requests that relate to other documents.

Http Request Headers

User agents (web browsers and other Http-enabled programs) include headers with the Http requests they send. Let's take another look at our sample Http request:

POST /Index.htm?sbd=2&lng=&thm=ProjectTheme&tzo=-660&ins=Index&xid=%7B7166F464-9484-4DE3-8F44-
                        F00CF7C9C74B%7D&sid=&cls=form&ViewMode=vmSingle&PageName=Home HTTP/1.1
Accept: */*
Accept-Language: de-at
Referer: http://www.morfik.com/
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; GTB6.3; SLCC1; .NET CLR 2.0.50727; 
           Media Center PC 5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729)
Host: www.morfik.com
Content-Length: 0
Connection: Keep-Alive
Pragma: no-cache 
Cookie: __utma=57227574.1756966090.1260398203.1260491327.1260491779.5; __utmb=57227574; 
            __utmz=57227574.1260398203.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); __utmc=57227574

[message body]

The values of these headers can be accessed using the THttpRequest class.

The Accept header specifies the media types that the user agent will accept in the response. It can be accessed as follows:

THttpRequest.Accept : String;

The Host header specifies the host name of the server. It can be accessed as follows:

THttpRequest.Host : String;

The If-Modified-Since header allows the client to specify that it only wants a fresh copy of the resource if it has been modified since a given date. It can be accessed as follows:

THttpRequest.IfModifiedSince : String;

The Referer header allows the client to specify the resource from which the request address was obtained. It can be accessed as follows:

THttpRequest.Referer : String;

The Accept-Encoding header indicates which compression algorithms are acceptable in the response:

THttpRequest.AcceptEncoding : String;   // e.g. 'gzip,deflate'

The Remote-Addr and Remote-Host headers identify the IP address of the client machine:

THttpRequest.RemoteAddress : String;    // IP address of the client machine
THttpRequest.RemoteHost    : String;    // host name of the client machine (blank if none available)

The User-Agent header identifies the program (usually a web browser) making the Http request. The value of this header can be obtained as follows:

THttpRequest.UserAgent : String;

This returns a string such as the following:

Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5

Rather than working with this string directly, it is usually easier to use the TBrowser class instead. This class includes a number of fields that represent specific pieces of information about the user agent.

The Engine and EngineID fields indicate the browser vendor:

(The Indy sockets library is an open-source library that supports Http and many other protocols. It is used in Morfik when calling a webservice published by another XApp.)

The Platform field of the TBrowser class indicates the operating system in use on the client machine:

THttpRequest.Browser.Platform : String;   // possible values: 'WIN', 'MAC', 'LNX'

The fields Version and MinorVersion indicate which version of the software is being used:

THttpRequest.Browser.Version      : Integer;   // e.g. 2009
THttpRequest.Browser.MinorVersion : Integer;   // e.g. 11

Finally, the Identifier field presents all five fields in a single string value:

THttpRequest.Browser.Identifier : String;   // e.g. GECKO(GK) 2009.11 WIN

Http Request Data

An Http request may have data attached. This could be, for example, xml code for a webservice call, or the content of a file that the user wishes to upload.

The data of an Http request may be accessed by means of the Buffer and BufferSize properties:

THttpRequest.Buffer     : Pointer;
THttpRequest.BufferSize : Integer;

Here is an example that shows how to save the xml code of a web service call to a file:

FX Code

Uses
    SystemCatalog;
 
Procedure WebMethod1.Execute;
Var
    MemoryStream : TMemoryStream;
Begin
    MemoryStream := TMemoryStream.Create();
    Try
        MemoryStream.WriteBuffer(SoapServer.Request.Buffer^, SoapServer.Request.BufferSize - 1);
        MemoryStream.SaveToFile(Catalog.RootDirectory + '\xml.xml');
    Finally
        MemoryStream.Free;
    End;
End;

Using SystemCatalog
 
Public Overrides Sub Execute
    Dim MemoryStream As TMemoryStream
    MemoryStream = New TMemoryStream()
    Try
        MemoryStream.WriteBuffer((Deref SoapServer.Request.Buffer), SoapServer.Request.BufferSize - 1)
        MemoryStream.SaveToFile(Catalog().RootDirectory & "\xml.xml")
    Finally
        MemoryStream.Free()
    End Try
End Sub

using SystemCatalog; 
 
public override void Execute()
{
    TMemoryStream MemoryStream;
    MemoryStream = new TMemoryStream();
    try
    {
        MemoryStream.WriteBuffer(*(SoapServer.Request.Buffer), SoapServer.Request.BufferSize - 1);
        MemoryStream.SaveToFile(Catalog().RootDirectory + "\\xml.xml");
    }
    finally
    {
        MemoryStream.Free();
    }
}

If the request data is known to be plain text, the following function may be used to obtain the request data as a string instead:

THttpRequest.GetAsText : String;

In some Http requests, the data contains multiple parts. Such Http requests have value 'multipart/form-data' in the Content-Type header. The sections are delimited by a special boundary string, whose value may be obtained from the following property:

THttpRequest.BoundaryString : String;

Http Response Headers

Http servers attach various headers to the Http response to provide information to the client. Let's take another look at our sample Http response:

HTTP/1.1 200 OK
Cache-Control: no-store
Allow: Post
Content-Type: text/html; charset=utf-8
Content-Encoding: deflate
Last-Modified: Fri, 11 Dec 2009 00:51:37 GMT
Server: Microsoft-IIS/7.0
Content: 
Date: Fri, 11 Dec 2009 00:51:37 GMT
Content-Length: 2807

[message body]

THttpResponse class in the Morfik framework provides various fields for setting these headers. Setting Http response headers is something that would most likely be done in the context of implementing a custom Http server.

The Http status code indicates whether the Http request was successful, and may provide further information about the status of the requested resource. The Http status code can be set using the following property:

THttpResponse.StatusCode : Integer;

Possible values include:

The Content-Encoding header specifies the encoding (compression method) used in the Http response data. It can be set using the following property:

THttpResponse.ContentEncoding : String;   // e.g. 'deflate'

The Content-Type header specifies the MIME type of the Http response data. It can be set using the following property:

THttpResponse.ContentType : String;   // e.g. 'text/xml'

The Content-Length header specifies the size of the Http response data in bytes. It can be set using the following property:

THttpResponse.ContentLength : Integer;

Finally, the value of any named Http response header can be set by use of the following function:

Procedure THttpResponse.AddHeader(Name,Value : String);

Http Response Data

An Http response may optionally include data, for example, the html code for a report, or a file that the user wishes to download.

The implementor of an Http server can use the following method to write data to the Http response:

Function THttpResponse.WriteBuffer(const Buf : Pointer; Count: Longint): Longint;

If the response is in plain text, the following methods may be used instead:

Procedure THttpResponse.WriteString  (const S : String);
Procedure THttpResponse.WritelnString(S : String);

The methods are also available in the THttpServer class:

Function  THttpServer.WriteBuffer(const Buf : Pointer; Count: Longint): Longint;
Procedure THttpServer.WriteString(S : String);
Procedure THttpServer.WritelnString(const S : String);

It is also possible to load response data from a file, using the following method:

Procedure THttpResponse.LoadFromFile(FileName : String);

Related Topics

• Looking under the hood of Morfik applications
• Understanding the key operations within the Morfik Application Framework
• URLs in a Morfik Application