ADC

Export transaction logs directly from NetScaler to Kafka

July 29, 2025

Contributed by:

Apache Kafka serves as a critical component in enterprise observability pipelines, offering a robust and highly efficient solution for managing logs. Instead of directly sending logs to observability platforms, Kafka provides capabilities to enhance the reliability and flexibility of log data flows.

The following are the advantages of exporting transaction logs to Kafka:

Ensures data integrity during outages.
Allows logs to be processed and refined before reaching the final observability tool.
Facilitates multiple consumers, thus enabling various systems to access the same log stream simultaneously for diverse analytical and operational needs.

Kafka broker diagram

Limitations

Configuring export of transaction logs using the NetScaler GUI is not supported.
A bootstrap server must respond with the FQDNs when queried by the NetScaler for the brokers for a topic. NetScaler must be able to resolve the FQDN’s of the broker.
Per topic, 1000 partitions are allowed across all brokers.
Kafka Authentication and Kafka encryption are not supported.
Kafka integration is not supported in NetScaler Cluster deployments.

Configure export of transaction logs to Kafka by using the NetScaler CLI

To export NetScaler transaction logs to Kafka by using the NetScaler CLI, perform the following steps:

Define a Kafka cluster.

add kafka cluster {Cluster Name}
<!--NeedCopy-->

Example:

add kafka cluster KafkaClusterProd
<!--NeedCopy-->

Create a service group of type KAFKA_BROKER and enable bootstrap attribute. This service group is used to connect to the bootstrap server to receive the metadata of the Kafka cluster. You can call this service group as “BootStrap Service Group”.
```
add servicegroup {BootStrap SG Name}  KAFKA_BROKER -bootstrap yes

```
Example:
```
add servicegroup KafkaBootstrapProd  KAFKA_BROKER -bootstrap yes

```
Create another service group of type KAFKA_BROKER and provide the topic to which NetScaler sends transaction logs. Ensure that autoscale is enabled and set to DNS. You can call this service group as “Topic Service Group”.
```
add servicegroup {Topic SG Name} KAFKA_BROKER -topicname {Topic Name} -autoscale DNS
 
```
Example:
```
add servicegroup KafkaTopicProd KAFKA_BROKER -topicname WebInsightProd -autoscale DNS

```

Bind both “BootStrap service group” and “Topic Service group” (created in the preceding steps) to the Kafka cluster.

bind kafka cluster {Cluster Name} -servicegroupname {BootStrap SG Name}
    
bind kafka cluster {Cluster Name} -servicegroupname {Topic SG Name}
<!--NeedCopy-->

Example:

bind kafka cluster KafkaClusterProd -servicegroupname KafkaBootstrapProd
    
bind kafka cluster KafkaClusterProd -servicegroupname KafkaTopicProd
<!--NeedCopy-->

Specify the FQDN and Port of the Kafka bootstrap server and bind it with the “BootStrap Service Group”. You must bind at least one broker to the bootstrap service group. Ensure that the FQDN specified in the bootstrap is resolvable by the NetScaler. NetScaler queries the bootstrap broker to get the metadata about the entire cluster, including the locations of other brokers and partitions for the “Topic”.
```
add server kafkacluster.broker0 <FQDN>
    
bind servicegroup {BootStrap SG Name} {Server name} {Port}

```
Example:
```
bind servicegroup KafkaBootstrapProd kafkacluster.broker0 9092

```

Enable the load balancing feature. Create a load balancing virtual server of type KAFKA_PRODUCER.

enable ns feature LB
    
add lb vserver {LB Name} KAFKA_PRODUCER -lbmethod ROUNDROBIN
<!--NeedCopy-->

Example:

enable ns feature LB

add lb vserver Kafka KAFKA_PRODUCER -lbmethod ROUNDROBIN
<!--NeedCopy-->

Only the round robin load balancing method is supported. If you try enabling any other load balancing method, you get an error. The following is a sample error:

> add lb vserver test KAFKA_PRODUCER -lbmethod LEASTCONNECTION
ERROR: Operation not permitted [Kafka Producer LB can use Round robin LB method only]
> add lb vserver test KAFKA_PRODUCER -lbmethod ROUNDROBIN
Warning: Feature(s) not enabled [LB]
Done
<!--NeedCopy-->

Bind the “Topic Service group” with the load balancing virtual server created in the preceding step.

bind lb vserver {LB Vserver Name} {Topic SG Name}
<!--NeedCopy-->

Example:

bind lb vserver Kafka KafkaTopicProd
<!--NeedCopy-->

Enable the AppFlow feature. Create an analytics profile and specify the collector as the load balancing virtual server created in the preceding steps.

enable ns feature appflow
add analytics profile {Analytics Profile Name} -collectors {LB Vserver Name} -type {Insight Type} -httpURL ENABLED -httpHost ENABLED -httpMethod ENABLED
<!--NeedCopy-->

Example:

enable ns feature appflow
add analytics profile WebInsightAnalytics -collectors Kafka -type webinsight -httpURL ENABLED -httpHost ENABLED -httpMethod ENABLED
<!--NeedCopy-->

Bind analytics profile to servers that you want to monitor.

bind lb vserver {LB Vserver Name} -analyticsProfile {Analytics Profile Name}
<!--NeedCopy-->

Example:

bind lb vserver ApplicationVserver -analyticsProfile WebInsightAnalytics
<!--NeedCopy-->

Monitor and Troubleshoot

Check if the Bootstrap service group and Topic service group are bound to the Kafka cluster and also check the status of the service.

> show kafka cluster KafkaClusterProd
        Name: KafkaClusterProd
        Bootstrap servicegroup: KafkaBootstrapProd
            Total Svc: 1        Active Svc: 1

        Topic Servicegroups:
        1)KafkaTopicProd
            Topic: WebInsightProd
            Total Svc: 2        Active Svc: 2
 Done
>
<!--NeedCopy-->

Check if the Kafka Bootstrap is enabled on the bootstrap service group.

> show servicegroup KafkaBootstrapProd
        KafkaBootstrapProd - KAFKA_BROKER
        State: ENABLED  Effective State: UP     Monitor Threshold : 0
        Max Conn: 0     Max Req: 0      Max Bandwidth: 0 kbits
        Use Source IP: NO
        Client Keepalive(CKA): NO
        Monitoring Owner: 0
        TCP Buffering(TCPB): NO
        HTTP Compression(CMP): NO
        Kafka Bootstrap: YES    Kafka TopicName: ???
        Idle timeout: Client: 180 sec   Server: 360 sec
        Client IP: DISABLED
        Cacheable: NO
        SC: ???
        SP: OFF
        Down state flush: ENABLED
        Monitor Connection Close : NONE
        Appflow logging: ENABLED
        ContentInspection profile name: ???
        Process Local: DISABLED
        Traffic Domain: 0


        1)     X.X.X.X:9092 State: UP       Server Name: kafkacluster.broker0       Server ID: None Weight: 1       Order: Default
                Last state change was at Tue Jul  8 04:44:50 2025
                Time since last state change: 0 days, 00:17:02.340

                Monitor Name: tcp-default       State: UP       Passive: 0
                Probes: 203     Failed [Total: 0 Current: 0]
                Last response: Success - TCP syn+ack received.
                Response Time: 0.000 millisec
 Done
>
<!--NeedCopy-->

Check if the Kafka topic name is set.

> show servicegroup KafkaTopicProd
        KafkaTopicProd - KAFKA_BROKER
        State: ENABLED  Effective State: UP     Monitor Threshold : 0
        Max Conn: 0     Max Req: 0      Max Bandwidth: 0 kbits
        Use Source IP: NO
        Client Keepalive(CKA): NO
        Monitoring Owner: 0
        TCP Buffering(TCPB): NO
        HTTP Compression(CMP): NO
        Kafka Bootstrap: ???    Kafka TopicName: WebInsightProd
        Idle timeout: Client: 180 sec   Server: 360 sec
        Client IP: DISABLED
        Cacheable: NO
        SC: ???
        SP: OFF
        Down state flush: ENABLED
        Monitor Connection Close : NONE
        Appflow logging: ENABLED
        Autoscale mode: DNS
        ContentInspection profile name: ???
        Process Local: DISABLED
        Traffic Domain: 0
        TROFS delayed movement: NO


        1)    Y.Y.Y.Y:9093 State: UP       Server Name: ns_broker1 (Auto scale)    Server ID: None Weight: 34      Order: Default
                Priority : 0
                Last state change was at Tue Jul  8 04:49:50 2025
                Time since last state change: 0 days, 00:13:07.890

                Monitor Name: tcp-default       State: UP       Passive: 0
                Probes: 156     Failed [Total: 0 Current: 0]
                Last response: Success - TCP syn+ack received.
                Response Time: 0.000 millisec

        2)  Z.Z.Z.Z:9094 State: UP       Server Name: ns_broker2 (Auto scale)    Server ID: None Weight: 33      Order: Default
                Priority : 0
                Last state change was at Tue Jul  8 04:49:50 2025
                Time since last state change: 0 days, 00:13:07.890

                Monitor Name: tcp-default       State: UP       Passive: 0
                Probes: 156     Failed [Total: 0 Current: 0]
                Last response: Success - TCP syn+ack received.
                Response Time: 0.000 millisec
 Done
>
<!--NeedCopy-->

Check if the topic service group is bound to KAFKA_PRODUCER load balancing virtual server.

> show lb vserver Kafka
        Kafka (0.0.0.0:0) - KAFKA_PRODUCER      Type: ADDRESS
        State: UP
        Last state change was at Tue Jul  8 04:49:50 2025
        Time since last state change: 0 days, 00:14:58.50
        Effective State: UP  ARP:DISABLED
        Client Idle Timeout: 180 sec
        Down state flush: ENABLED
        Disable Primary Vserver On Down : DISABLED
        Appflow logging: ENABLED
        No. of Bound Services :  2 (Total)       2 (Active)
        Configured Method: ROUNDROBIN
        Current Method: Round Robin, Reason: A new service is bound     BackupMethod: NONE
        Mode: IP
        Persistence: NONE
        L2Conn: OFF
        Skip Persistency: None
        Listen Policy: NONE
        IcmpResponse: PASSIVE
        RHIstate: PASSIVE
        New Service Startup Request Rate: 0 PER_SECOND, Increment Interval: 0
        Mac mode Retain Vlan: DISABLED
        DBS_LB: DISABLED
        Process Local: DISABLED
        Traffic Domain: 0
        TROFS Persistence honored: ENABLED
        Retain Connections on Cluster: NO
        Order Sequence: ASCENDING
        Current Active Order: None

Bound Service Groups:
1)      Group Name: KafkaTopicProd

                1) KafkaTopicProd (Y.Y.Y.Y: 9093) -  State: UP     Weight: 34 Order: None
                2) KafkaTopicProd (Z.Z.Z.Z: 9094) -  State: UP   Weight: 33 Order: None
 Done
>
<!--NeedCopy-->

The following table provides the details of supported Kafka counters.

Counter	Description
kafka_tot_produce_request_sent	Total Produce requests sent
kafka_tot_produce_response_rcvd	Total Produce responses received
kafka_tot_produce_resp_success	Total successful produce responses
kafka_err_produce_response	Total error produce responses
kafka_err_produce_resp_err_code	Error code value of Kafka produce response
kafka_err_produce_request	Total error produce requests
kafka_tot_metadata_request_sent	Total metadata requests sent
kafka_tot_metadata_response_rcvd	Total successful metadata responses received
kafka_err_metadata_response	Total error metadata responses
kafka_err_metadata_request	Total error metadata requests
kafka_err_metadata_resp_err_code	Error code value of Kafka metadata response

The official version of this content is in English. Some of the Cloud Software Group documentation content is machine translated for your convenience only. Cloud Software Group has no control over machine-translated content, which may contain errors, inaccuracies or unsuitable language. No warranty of any kind, either expressed or implied, is made as to the accuracy, reliability, suitability, or correctness of any translations made from the English original into any other language, or that your Cloud Software Group product or service conforms to any machine translated content, and any warranty provided under the applicable end user license agreement or terms of service, or any other agreement with Cloud Software Group, that the product or service conforms with any documentation shall not apply to the extent that such documentation has been machine translated. Cloud Software Group will not be held responsible for any damage or issues that may arise from using machine-translated content.

Was this helpful

NetScaler Secure Deployment Guide

Export transaction logs directly from NetScaler to Kafka

July 29, 2025

Contributed by:

July 29, 2025

Contributed by:

Export transaction logs directly from NetScaler to Kafka

Limitations

Configure export of transaction logs to Kafka by using the NetScaler CLI

Monitor and Troubleshoot

In this article