Protocol extensions - use cases

Protocol extensions can be used for the following use cases.

• Message based load balancing (MBLB)
• Streaming
• Token based load balancing
• Load balancing persistence
• TCP connection based load balancing
• Content based load balancing
• SSL
• Modify traffic
• Originate traffic to client or server
• Process data on the connection establishment

Message based load balancing

Protocol extensions support Message Based Load Balancing (MBLB), which can parse any protocol on a NetScaler appliance and load balance the protocol messages arriving on one client connection, that is, distribute the messages over multiple server connections. MBLB is achieved by user code that parses the client TCP data stream.

The TCP data stream is passed to the on_data callbacks for client and server behaviors. The TCP data stream is available to the extension functions through a Lua string like interface. You can use an API similar to the Lua string API to parse the TCP data stream.

Useful APIs include:

data:len()

data:find()

data:byte()

data:sub()

data:split()

Once the TCP data stream has been parsed into a protocol message, the user code achieves load balancing by just sending the protocol message to the next context available from the context passed to the on_data callback for the client.

The ns.send() API is used to send messages to other processing modules. In addition to the destination context, the send API takes the event name and optional payload as arguments. There is one-to-one correspondence between the event name and the callback function names for the behaviors. The callbacks for events are called on_<event_name>. The callback names use only lowercase.

For example, the TCP client and server on_data callbacks are user-defined handlers for events named “DATA.” For sending the whole protocol message in one send call, the EOM event is used. EOM, which stands for end of message, signifies the end of protocol message to the LB context down stream, so a new load balancing decision is made for data that follows this message.

The extension code might sometimes not receive the whole protocol message in the on_data event. In such a case, the data can be held by using the ctxt:hold() API. The hold API is available for both TCP-client and server-callback contexts. When “hold with data” is called, the data is stored in the context. When more data is received in the same context, the newly received data is appended to the previously stored data and the on_data callback function is called again with the combined data.

Note: The load balancing method used depends on the configuration of the load balancing virtual server corresponding to the load balancing context.

The following code snippet shows the use of the send API to send the parsed protocol message.

Example:

    function client.on_data(ctxt, payload)
        --
        -- code to parse payload.data into protocol message comes here
        --
        -- sending the message to lb
        ns.send(ctxt.output, "EOM", {data = message})
    end -- client.on_data

    function server.on_data(ctxt, payload)
        --
        -- code to parse payload.data into protocol message comes here
        --
        -- sending the message to client
        ns.send(ctxt.output, "EOM", {data = message})

    end -- server.on_data
<!--NeedCopy-->

Streaming

In some scenarios, holding the TCP data stream until the whole protocol message is collected might not be necessary. In fact, it is not advised unless it is required. Holding the data increases memory usage on NetScaler appliance and can make the appliance susceptible to DDoS attacks by exhausting the memory on NetScaler appliance with incomplete protocol messages on many connections.

Users can achieve streaming of TCP data in the extension callback handlers by using the send API. Instead of holding the data until the whole message is collected, data can be sent in chunks. Sending data to ctxt.output by using the DATA event sends a partial protocol message. It can be followed by more DATA events. An EOM event must be sent to mark the end of the protocol message. The load balancing context downstream makes the load balancing decision on the first data received. A new load balancing decision is made after the receipt of the EOM message.

To stream protocol message data, send multiple DATA events followed by an EOM event. The contiguous DATA events and the following EOM event are sent to the same server connection selected by load balancing decision for the first DATA event in the sequence.

For a send to client context, EOM and DATA events are effectively the same, because there is no special handling by the client context downstream for EOM events.

Token based load balancing

For natively supported protocols, a NetScaler appliance supports a token based load balancing method that uses PI expressions to create the token. For extensions, the protocol is not known in advance, so PI expressions cannot be used. For token based load balancing, you have to set the default load balancing virtual server to use the USER_TOKEN load balancing method, and provide the token value from the extension code by calling the send API with a user_token field. If the token value is sent from the send API and the USER_TOKEN load balancing method is configured on the default load balancing virtual server, the load balancing decision is made by calculating a hash based on the token value. The maximum length of token value is 64 bytes.

add lb vserver v\_mqttlb USER\_TCP –lbMethod USER\_TOKEN

The code snippet in the following example uses a send API to send an LB token value.

Example:

        -- send the message to lb




        -- user_token is set to do LB based on clientID




        ns.send(ctxt.output, "EOM", {data = message,

                                 user_token = token_info})
<!--NeedCopy-->

Load balancing persistence

Load balancing persistence is closely related to token based load balancing. Users have to be able to programmatically calculate the persistence session value and use it for load balancing persistence. The send API is used to send persistence parameters. To use load balancing persistence, you have to set the USERSESSION persistence type on the default load balancing virtual server and provide a persistence parameter from the extension code by calling the send API with a user_session field. The maximum length of the persistence parameter value is 64 bytes.

If you need multiple types of persistence for a custom protocol, you have to define user persistence types and configure them. The names of the parameters used to configure the virtual servers are decided by the protocol implementer. A parameter’s configured value is also available to the extension code.

The following CLI and code snippet shows the use of a send API to support load balancing persistence. The code listing in the section Code Listing for mqtt.lua also illustrates the use of the user_session field.

For persistency, you have to specify the USERSESSION persistency type on the load balancing virtual server and pass the user_session value from the ns.send API.

add lb vserver v\_mqttlb USER\_TCP –persistencetype USERSESSION

Send the MQTT message to the load balancer, with the user_session field set to clientID in the payload.

Example:

-- send the data so far to lb

-- user_session is set to clientID as well (it will be used to persist session)

ns.send(ctxt.output, “DATA”, {data = data, user_session = clientID})
<!--NeedCopy-->

TCP connection based load balancing

For some protocols, MBLB might not be needed. Instead, you might need TCP connection based load balancing. For example, the MQTT protocol must parse the initial part of the TCP stream to determine the token for load balancing. And, all the MQTT messages on the same TCP connection must be sent to the same server connection.

TCP connection based load balancing can be achieved by using the send API with only DATA events and not sending any EOM. That way the downstream load balancing context bases the load balancing decision on the data received first, and sends all the subsequent data to the same server connection selected by the load balancing decision.

Also, some use cases might require the ability to bypass extension handling after the load balancing decision has been made. Bypassing the extension calls results in better performance, because the traffic is processed purely by native code. Bypass can be done by using the ns.pipe() API. A call to the pipe() API extension code can connect input context to an output context. After the call to pipe(), all the events coming from input context directly go to the output context. Effectively, the module from which the pipe() call is made is removed from the pipeline.

The following code snippet shows streaming and the use of the pipe() API to bypass a module. The code listing in the section Code Listing for mqtt.lua also illustrates how to do streaming and the use of pipe() API to bypass the module for rest of the traffic on the connection.

Example:

        -- send the data so far to lb
        ns.send(ctxt.output, "DATA", {data = data,
                                       user_token = clientID})
        -- pipe the subsequent traffic to the lb - to bypass the client on_data handler
        ns.pipe(ctxt.input, ctxt.output)
<!--NeedCopy-->

Content based load balancing

For native protocols, content switching like feature for protocol extensions is supported. With this feature, instead of sending the data to the default load balance, you can send the data to the selected load balancer.

Content switching feature for protocol extensions is achieved by using the ctxt:lb_connect(<lbname>) API. This API is available to the TCP client context. Using this API, the extension code can obtain a load balancing context corresponding to an already configured load balancing virtual server. You can then use the send API with the load balancing context thus obtained.

The lb context can be NULL sometimes:

• Virtual server does not exists
• Virtual server is not of user protocol type
• Virtual server’s state is not UP
• Virtual server is user virtual server, not load balancing virtual server

If you remove the target load balancing virtual server when it is in use, then all connections associated with that load balancing virtual server is reset.

The following code snippet shows the use of lb_connect() API. The code maps the client ID to load balancing virtual server names (lbname) using the Lua table lb_map and then gets the LB context for lbname using lb_connect(). And finally sends to the LB context using send API.

    local lb_map = {
       ["client1*"] = "lb_1",
       ["client2*"] = "lb_2",
       ["client3*"] = "lb_3",
       ["client4*"] = "lb_4"
    }

    -- map the clientID to the corresponding LB vserver and connect to it
    for client_pattern, lbname in pairs(lb_map) do
       local match_idx = string.find(clientID, client_pattern)
       if (match_idx == 1) then
      lb_ctxt = ctxt:lb_connect(lbname)
      if (lb_ctxt == nil) then
         error("Failed to connect to LB vserver: " .. lbname)
      end
      break
       end
    end
    if (lb_ctxt == nil) then
    -- If lb context is NULL, the user can raise an error or send data to default LB
       error("Failed to map LB vserver for client: " .. clientID)
    end
-- send the data so far to lb
ns.send(lb_ctxt, "DATA", {data = data}
<!--NeedCopy-->

SSL

SSL for protocols using extensions is supported in ways similar to how SSL for native protocols is supported. Using the same parsing code for creating custom protocols, you can create a protocol instance over TCP or over SSL which can then be used to configure the virtual servers. Similarly, you can add user services over TCP or SSL.

For more information, see Configuring SSL Offloading for MQTT and Configuring SSL Offloading for MQTT With End-To-End Encryption.

Server connection multiplexing

Sometimes, the client sends one request at a time and sends the next request only after the response for the first request is received from the server. In such a case, server connection can be reused for other client connections, and for the next message on the same connection, after the response has been sent to the client. To allow reuse of server connection by other client connections, you must use the ctxt: reuse_server_connection() API on the server side context.

Note: This API is available in NetScaler 12.1 build 49.xx and later.

Modify traffic

To modify data in the request or response, you must use the native rewrite feature that uses an advanced policy PI expression. Because you cannot use PI expressions in extensions, you can use the following APIs to modify a TCP stream data.

data:replace(offset, length, new_string)
data:insert(offset, new_string)
data:delete(offset, length)
data:gsub(pattern, replace [,n]))

The following code snippet shows the use of replace() API.

-- Get the offset of the pattern, we want to replace
   local old_pattern = “pattern to repalace”
local old_pattern_length = old_pattern:len()
   local pat_off, pat_end = data:find(old_pattern)
   -- pattern is not present
if (not pat_off) then
    goto send_data
   end
  -- If the data we want to modify is not completely present, then
  -- wait for more data
  if (not pat_end) then
        ctxt:hold(data)
        data = nil
     goto done
  end
data:replace(pat_off, old_pattern_length, “new pattern”)
::send_data::
ns.send(ctxt.output, “EOM”, {data = data})
::done::

The following code snippet shows the use of insert() API.

data:insert(5, “pattern to insert”)

The following code snippet shows the use of insert() API, when we want to insert after or before some pattern:

-- Get the offset of the pattern, after or before which we want to insert
   local pattern = “pattern after/before which we need to insert”
local pattern_length = pattern:len()
   local pat_off, pat_end = data:find(pattern)
-- pattern is not present
   if (not pat_off) then
    goto send_data
   end
  -- If the pattern after which we want to insert is not
  -- completely present, then wait for more data
  if (not pat_end) then
        ctxt:hold(data)
        data = nil
     goto done
  end
-- Insert after the pattern
data:insert(pat_end + 1, “pattern to insert”)
   -- Insert before the pattern
data:insert(pat_off, “pattern to insert”)
::send_data::
    ns.send(ctxt.output, “EOM”, {data = data})
::done::

The following code snippet shows the use of delete() API.

-- Get the offset of the pattern, we want to delete
   local delete_pattern = “pattern to delete”
local delete_pattern_length = delete_pattern:len()
   local pat_off, pat_end = data:find(old_pattern)
   -- pattern is not present
if (not pat_off) then
          goto send_data
   end
  -- If the data we want to delete is not completely present,
  -- then wait for more data
  if (not pat_end) then
        ctxt:hold(data)
        data = nil
    goto done
  end
data:delete(pat_off, delete_pattern_length)
::send_data::
ns.send(ctxt.output, “EOM”, {data = data})
::done::

The following code snippet shows the use of gsub() API.

    -- Replace all the instances of the pattern with the new string
data:gsub(“old pattern”, “new string”)
-- Replace only 2 instances of “old pattern”
data:gsub(“old pattern”, “new string”, 2)
-- Insert new_string before all instances of “http”
data:gsub(“input data”, “(http)”, “new_string%1”)
-- Insert new_string after all instances of “http”
data:gsub(“input data”, “(http)”, “%1new_string”)
-- Insert new_string before only 2 instances of “http”
data:gsub(“input data”, “(http)”, “new_string%1”, 2)

Note: This API is available in NetScaler 12.1 build 50.xx and later.

Originate traffic to client or server

You can use the ns.send() API to send data that originates from the extension code to a client and a back-end server. To send or receive response directly with a client, from client context, you must use ctxt.client as the target. To send or receive response directly with a back-end server from server context, you must use ctxt.server as the target. The data in the payload can be a TCP stream data or a Lua string.

To stop traffic processing on a connection, you can use ctxt:close() API from either the client or the server context. This API closes the client-side connection or any server connections linked to it.

When you call the ctxt:close() API, extension code sends TCP FIN packet to the client and server connections and if more data is received from the client or server on this connection, then the appliance resets the connection.

The following code snippet shows the use of ctxt.client and ctxt:close() APIs.

    -- If the input packet is not MQTT CONNECT type, then
-- send some error response to the client.
function client.on_data(ctxt, payload)
    local data = payload.data
    local offset = 1
    local msg_type = 0
    local error_response = “Missing MQTT Connect packet.”
    byte = data:byte(offset)
msg_type = bit32.rshift(byte, 4)
if (msg_type ~= 1) then
-- Send the error response
   ns.send(ctxt.client, “DATA”, {data = error_response})
-- Since error response has been sent, so now close the connection
    ctxt:close()
end

The following code snippet shows the example when user can inject the data in the normal traffic flow.

-- After sending request, send some log message to the server.
function client.on_data(ctxt, payload)
local data = payload.data
local log_message = “client id : “..data:sub(3, 7)..” user name : “ data:sub(9, 15)
-- Send the request we get from the client to backend server
ns.send(ctxt.output, “DATA”, {data = data})
After sending the request, also send the log message
ns.send(ctxt.output, “DATA”, {data = log_message”})
end

The following code snippet shows the use of ctxt.to_server API.

-- If the HTTP response status message is “Not Found”,
-- then send another request to the server.
function server.on_data(ctxt, payload)
    local data = payload.data
    local request “GET /default.html HTTP/1.1\r\n\r\n”ss
    local start, end = data:find(“Not Found”)
    if (start) then
    -- Send the another request to server
        ns.send(ctxt.server, “DATA”, {data = request})
end

Note: This API is available in NetScaler 12.1 build 50.xx and later.

Data processing on the connection establishment

There might be a use case where you want to send some data at the connection establishment (when the final ACK is received). For example, in proxy protocol, you might want to send client’s source and destination IP addresses and ports to the back-end server at the connection establishment. In this case, you can use client.init() callback handler to send the data on connection establishment.

The following code snippet shows the use of client.init() callback:

-- Send a request to the next processing context
-- on the connection establishment.
function client.init(ctxt)
   local request “PROXY TCP4” + ctxt.client.ip.src.to_s + “ “ + ctxt.client.ip.dst.to_s + “ “ + ctxt.client.tcp.srcport + “ “ +     ctxt.client.tcp.dstport
-- Send the another request to server
   ns.send(ctxt.output, “DATA”, {data = request})
  end

Note: This API is available in NetScaler 13.0 build xx.xx and later.