You are viewing docs on Elastic's new documentation system, currently in technical preview. For all other Elastic docs, visit elastic.co/guide.
Last updated: Apr 10th, 2023

Zscaler Internet Access

Collect logs from Zscaler Internet Access (ZIA) with Elastic Agent.

What is an Elastic integration?

This integration is powered by Elastic Agent. Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, forward data from remote services or hardware, and more. Refer to our documentation for a detailed comparison between Beats and Elastic Agent.

Prefer to use Beats for this use case? See Filebeat modules for logs or Metricbeat modules for metrics.

This integration is for Zscaler Internet Access logs. It can be used to receive logs sent by NSS log server on respective TCP ports.

The log message is expected to be in JSON format. The data is mapped to ECS fields where applicable and the remaining fields are written under zscaler_zia.<data-stream-name>.*.

Steps for setting up NSS Feeds

  1. Enable the integration with the TCP input.
  2. Configure the Zscaler NSS Server and NSS Feeds to send logs to the Elastic Agent that is running this integration. See Add NSS Server and Add NSS Feeds. Use the IP address hostname of the Elastic Agent as the 'NSS Feed SIEM IP Address/FQDN', and use the listening port of the Elastic Agent as the 'SIEM TCP Port' on the Add NSS Feed configuration screen. To configure Zscaler NSS Server and NSS Feeds follow the following steps.
    • In the ZIA Admin Portal, add an NSS Server.
      • Log in to the ZIA Admin Portal using your admin account. If you're unable to log in, contact Support.
      • Add an NSS server. Refer to Adding NSS Servers to set up an Add NSS Server for Web and/or Firewall.
      • Verify that the state of the NSS Server is healthy.
        • In the ZIA Admin Portal, go to Administration > Nanolog Streaming Service > NSS Servers.
        • In the State column, confirm that the state of the NSS server is healthy.
    • In the ZIA Admin Portal, add an NSS Feed.
      • Refer to Add NSS Feeds and select the type of feed you want to configure. The following fields require specific inputs:
        • SIEM IP Address: Enter the IP address of the Elastic agent you’ll be assigning the Zscaler integration to.
        • SIEM TCP Port: Enter the port number, depending on the logs associated with the NSS Feed. You will need to create an NSS Feed for each log type.
          • Alerts: 9010
          • DNS: 9011
          • Firewall: 9012
          • Tunnel: 9013
          • Web: 9014
        • Feed Output Type: Select Custom in Feed output type and paste the appropriate response format in Feed output format as follows:

Steps for setting up Cloud NSS Feeds

  1. Enable the integration with the HTTP Endpoint input.
  2. Configure the Zscaler Cloud NSS Feeds to send logs to the Elastic Agent that is running this integration. Provide API URL to send logs to the Elastic Agent. To configure Zscaler Cloud NSS Feeds follow the following steps.
    • In the ZIA Admin Portal, add a Cloud NSS Feed.
      • Log in to the ZIA Admin Portal using your admin account.
      • Add a Cloud NSS Feed. See to Add Cloud NSS Feed.
        • In the ZIA Admin Portal, go to Administration > Nanolog Streaming Service > Cloud NSS Feeds.
        • Give Feed Name, change status to Enabled.
        • Select NSS Type.
        • Change SIEM Type to other.
        • Add an API URL.
        • Default ports:
          • DNS: 9556
          • Firewall: 9557
          • Tunnel: 9558
          • Web: 9559
        • Select JSON as feed output type.
        • Add same custom header along with its value on both the side for additional security.
  3. Repeat step 2 for each log type.

Please make sure to use the given response formats for NSS and Cloud NSS Feeds.

Note: Please make sure to use latest version of given response formats.

Compatibility

This package has been tested against Zscaler Internet Access version 6.1

Documentation and configuration

Alerts

  • Default port (NSS Feed): 9010

See: Zscaler Vendor documentation

Zscaler response format (v1):

<%d{syslogid}>%s{Monthname} %2d{Dayofmonth} %02d{Hour}:%02d{Minutes}:%02d{Seconds} [%s{Deviceip}] ZscalerNSS: %s{Eventinfo}\n

Sample Response:

<114>Dec 10 14:04:28 [175.16.199.1] ZscalerNSS: Zscaler cloud configuration connection to  175.16.199.1:443 lost and unavailable for the past 2325.00 minutes

DNS Log

  • Default port (NSS Feed): 9011
  • Default port (Cloud NSS Feed): 9556

See: Zscaler Vendor documentation

Zscaler response format (v1):

\{ "sourcetype" : "zscalernss-dns", "event" :\{"datetime":"%s{time}","user":"%s{elogin}","department":"%s{edepartment}","location":"%s{elocation}","reqaction":"%s{reqaction}","resaction":"%s{resaction}","reqrulelabel":"%s{reqrulelabel}","resrulelabel":"%s{resrulelabel}","dns_reqtype":"%s{reqtype}","dns_req":"%s{req}","dns_resp":"%s{res}","srv_dport":"%d{sport}","durationms":"%d{durationms}","clt_sip":"%s{cip}","srv_dip":"%s{sip}","category":"%s{domcat}","deviceowner":"%s{deviceowner}","devicehostname":"%s{devicehostname}"\}\}

Sample Response:

{ "sourcetype" : "zscalernss-dns", "event" :{"datetime":"Fri Dec 17 07:27:54 2021","user":"some_user@example.com","department":"Unknown","location":"TestLoc%20DB","reqaction":"REQ_ALLOW","resaction":"Some Response Action","reqrulelabel":"Access%20Blocked","resrulelabel":"None","dns_reqtype":"Some type","dns_req":"example.com","dns_resp":"Some response string","srv_dport":"8080","durationms":"123456","clt_sip":"81.2.69.193","srv_dip":"81.2.69.144","category":"Professional Services","deviceowner":"Owner77","devicehostname":"Machine9000"}}

Firewall Log

  • Default port (NSS Feed): 9012
  • Default port (Cloud NSS Feed): 9557

See: Zscaler Vendor documentation

Zscaler response format (v1):

\{ "sourcetype" : "zscalernss-fw", "event" :\{"datetime":"%s{time}","user":"%s{elogin}","department":"%s{edepartment}","locationname":"%s{elocation}","cdport":"%d{cdport}","csport":"%d{csport}","sdport":"%d{sdport}","ssport":"%d{ssport}","csip":"%s{csip}","cdip":"%s{cdip}","ssip":"%s{ssip}","sdip":"%s{sdip}","tsip":"%s{tsip}","tunsport":"%d{tsport}","tuntype":"%s{ttype}","action":"%s{action}","dnat":"%s{dnat}","stateful":"%s{stateful}","aggregate":"%s{aggregate}","nwsvc":"%s{nwsvc}","nwapp":"%s{nwapp}","proto":"%s{ipproto}","ipcat":"%s{ipcat}","destcountry":"%s{destcountry}","avgduration":"%d{avgduration}","rulelabel":"%s{erulelabel}","inbytes":"%ld{inbytes}","outbytes":"%ld{outbytes}","duration":"%d{duration}","durationms":"%d{durationms}","numsessions":"%d{numsessions}","ipsrulelabel":"%s{ipsrulelabel}","threatcat":"%s{threatcat}","threatname":"%s{ethreatname}","deviceowner":"%s{deviceowner}","devicehostname":"%s{devicehostname}"\}\}

Sample Response:

{ "sourcetype" : "zscalernss-fw", "event" :{"datetime":"Fri Dec 17 07:27:54 2021","user":"some_user@example.com","department":"Unknown","locationname":"TestLoc%20DB","cdport":443,"csport":55018,"sdport":443,"ssport":0,"csip":"0.0.0.0","cdip":"0.0.0.0","ssip":"0.0.0.0","sdip":"0.0.0.0","tsip":"0.0.0.0","tunsport":0,"tuntype":"ZscalerClientConnector","action":"Drop","dnat":"No","stateful":"Yes","aggregate":"No","nwsvc":"HTTPS","nwapp":"http","proto":"TCP","ipcat":"Test Name","destcountry":"Ireland","avgduration":486,"rulelabel":"Access%20Blocked","inbytes":19052,"outbytes":1734,"duration":0,"durationms":486,"numsessions":1,"ipsrulelabel":"None","threatcat":"None","threatname":"None","deviceowner":"admin77","devicehostname":"Machine9000"}}

Tunnel Log

  • Default port (NSS Feed): 9013
  • Default port (Cloud NSS Feed): 9558

See: Zscaler Vendor documentation

Zscaler response format (v1):

  • Tunnel Event: 
    \{ "sourcetype" : "zscalernss-tunnel", "event" : \{"datetime":"%s{datetime}","Recordtype":"%s{tunnelactionname}","tunneltype":"%s{tunneltype}","user":"%s{vpncredentialname}","location":"%s{elocationname}","sourceip":"%s{sourceip}","destinationip":"%s{destvip}","sourceport":"%d{srcport}","event":"%s{event}","eventreason":"%s{eventreason}","recordid":"%d{recordid}"\}\}
  • Sample Event: 
    \{ "sourcetype" : "zscalernss-tunnel", "event" : \{"datetime":"%s{datetime}","Recordtype":"%s{tunnelactionname}","tunneltype":"%s{tunneltype}","user":"%s{vpncredentialname}","location":"%s{elocationname}","sourceip":"%s{sourceip}","destinationip":"%s{destvip}","sourceport":"%d{srcport}","txbytes":"%lu{txbytes}","rxbytes":"%lu{rxbytes}","dpdrec":"%d{dpdrec}","recordid":"%d{recordid}"\}\}
  • IKE Phase 1 
    \{ "sourcetype" : "zscalernss-tunnel", "event" : \{"datetime":"%s{datetime}","Recordtype":"%s{tunnelactionname}","tunneltype":"IPSEC IKEV %d{ikeversion}","user":"%s{vpncredentialname}","location":"%s{elocationname}","sourceip":"%s{sourceip}","destinationip":"%s{destvip}","sourceport":"%d{srcport}","destinationport":"%d{dstport}","lifetime":"%d{lifetime}","ikeversion":"%d{ikeversion}","spi_in":"%lu{spi_in}","spi_out":"%lu{spi_out}","algo":"%s{algo}","authentication":"%s{authentication}","authtype":"%s{authtype}","recordid":"%d{recordid}"\}\}
  • IKE Phase 2 
    \{ "sourcetype" : "zscalernss-tunnel", "event" : \{"datetime":"%s{datetime}","Recordtype":"%s{tunnelactionname}","tunneltype":"IPSEC IKEV %d{ikeversion}","user":"%s{vpncredentialname}","location":"%s{elocationname}","sourceip":"%s{sourceip}","destinationip":"%s{destvip}","sourceport":"%d{srcport}","sourceportstart":"%d{srcportstart}","destinationportstart":"%d{destportstart}","srcipstart":"%s{srcipstart}","srcipend":"%s{srcipend}","destinationipstart":"%s{destipstart}","destinationipend":"%s{destipend}","lifetime":"%d{lifetime}","ikeversion":"%d{ikeversion}","lifebytes":"%d{lifebytes}","spi":"%d{spi}","algo":"%s{algo}","authentication":"%s{authentication}","authtype":"%s{authtype}","protocol":"%s{protocol}","tunnelprotocol":"%s{tunnelprotocol}","policydirection":"%s{policydirection}","recordid":"%d{recordid}"\}\}

Sample Response:

{ "sourcetype" : "zscalernss-tunnel", "event" : {"datetime":"Thu Dec 30 11:40:27 2021","Recordtype":"IPSec Phase1","tunneltype":"IPSEC IKEV 2","user":"81.2.69.145","location":"some-location","sourceip":"81.2.69.145","destinationip":"81.2.69.143","sourceport":"500","destinationport":"500","lifetime":"0","ikeversion":"2","spi_in":"00000000000000000000","spi_out":"11111111111111111111","algo":"AES-CBS","authentication":"HMAC-SHA1-96","authtype":"PSK","recordid":"1111111111111111111"}}

Web Log

  • Default port (NSS Feed): 9014
  • Default port (Cloud NSS Feed): 9559
  • Add characters " and \ in feed escape character while configuring Web Log.


See: Zscaler Vendor documentation

Zscaler response format (v2):

\{ "sourcetype" : "zscalernss-web", "event" :\{"time":"%s{time}","login":"%s{login}","proto":"%s{proto}","eurl":"%s{eurl}","action":"%s{action}","appname":"%s{appname}","appclass":"%s{appclass}","reqsize":"%d{reqsize}","respsize":"%d{respsize}","stime":"%d{stime}","ctime":"%d{ctime}","urlclass":"%s{urlclass}","urlsupercat":"%s{urlsupercat}","urlcat":"%s{urlcat}","malwarecat":"%s{malwarecat}","threatname":"%s{threatname}","riskscore":"%d{riskscore}","dlpeng":"%s{dlpeng}","dlpdict":"%s{dlpdict}","location":"%s{location}","dept":"%s{dept}","cip":"%s{cip}","sip":"%s{sip}","reqmethod":"%s{reqmethod}","respcode":"%s{respcode}","eua":"%s{eua}","ereferer":"%s{ereferer}","ruletype":"%s{ruletype}","rulelabel":"%s{rulelabel}","contenttype":"%s{contenttype}","unscannabletype":"%s{unscannabletype}","deviceowner":"%s{deviceowner}","devicehostname":"%s{devicehostname}"\}\}

Sample Response:

{ "sourcetype" : "zscalernss-web", "event" :{"time":"Fri Dec 17 07:04:57 2021","login":"test@example.com","proto":"HTTP_PROXY","eurl":"browser.events.data.msn.com:443","action":"Blocked","appname":"General Browsing","appclass":"General Browsing","reqsize":"600","respsize":"65","stime":"0","ctime":"0","urlclass":"Business Use","urlsupercat":"Information Technology","urlcat":"Web Search","malwarecat":"None","threatname":"None","riskscore":"0","dlpeng":"None","dlpdict":"None","location":"Test DB","dept":"Unknown","cip":"81.2.69.193","sip":"81.2.69.145","reqmethod":"CONNECT","respcode":"200","eua":"Windows%20Microsoft%20Windows%2010%20Pro%20ZTunnel%2F1.0","ereferer":"None","ruletype":"FwFilter","rulelabel":"Zscaler Proxy Traffic","contenttype":"Other","unscannabletype":"None","deviceowner":"administrator1","devicehostname":"TestMachine35"}}

Caveats:

  • To ensure that URLs are processed correctly, logs which have a network.protocol value that is not http or https will be implicitly converted to https for the purposes of URL parsing. The original value of network.protocol will be preserved.

Logs reference

alerts

This is the alerts dataset.

Example

An example event for alerts looks as following:

{
    "@timestamp": "2023-12-10T13:40:32.000Z",
    "agent": {
        "ephemeral_id": "19a4d9cc-161f-44b9-803e-3d5f3f7668d7",
        "id": "fc4affb9-ab52-48ec-b9ce-f65f4390f0b9",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.6.0"
    },
    "data_stream": {
        "dataset": "zscaler_zia.alerts",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "address": "81.2.69.193",
        "ip": "81.2.69.193",
        "port": 9012
    },
    "ecs": {
        "version": "8.7.0"
    },
    "elastic_agent": {
        "id": "fc4affb9-ab52-48ec-b9ce-f65f4390f0b9",
        "snapshot": false,
        "version": "8.6.0"
    },
    "event": {
        "agent_id_status": "verified",
        "dataset": "zscaler_zia.alerts",
        "ingested": "2023-02-24T09:40:42Z"
    },
    "input": {
        "type": "tcp"
    },
    "log": {
        "source": {
            "address": "172.29.0.7:55310"
        },
        "syslog": {
            "priority": 114
        }
    },
    "message": "ZscalerNSS: SIEM Feed connection \"DNS Logs Feed\" to 81.2.69.193:9012 lost and unavailable for the past 2440.00 minutes",
    "related": {
        "ip": [
            "81.2.69.193"
        ]
    },
    "tags": [
        "forwarded",
        "zscaler_zia-alerts"
    ],
    "zscaler_zia": {
        "alerts": {
            "connection_lost_minutes": 2440,
            "log_feed_name": "DNS Logs Feed"
        }
    }
}

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset.
constant_keyword
event.module
Event module.
constant_keyword
input.type
Type of Filebeat input.
keyword
log.offset
Log offset.
long
log.source.address
Source address from which the log event was read / sent from.
keyword
tags
User defined tags.
keyword
zscaler_zia.alerts.connection_lost_minutes
Amount of time after loosing connection to a server in Minutes.
double
zscaler_zia.alerts.log_feed_name
Name of the NSS log feed.
keyword

dns

This is the dns dataset.

Example

An example event for dns looks as following:

{
    "@timestamp": "2021-12-17T07:27:54.000Z",
    "agent": {
        "ephemeral_id": "f4bda282-f29f-4110-ae6f-3e625eb7e71b",
        "id": "fc4affb9-ab52-48ec-b9ce-f65f4390f0b9",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.6.0"
    },
    "data_stream": {
        "dataset": "zscaler_zia.dns",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "as": {
            "number": 29518,
            "organization": {
                "name": "Bredband2 AB"
            }
        },
        "geo": {
            "city_name": "Linköping",
            "continent_name": "Europe",
            "country_iso_code": "SE",
            "country_name": "Sweden",
            "location": {
                "lat": 58.4167,
                "lon": 15.6167
            },
            "region_iso_code": "SE-E",
            "region_name": "Östergötland County"
        },
        "ip": "89.160.20.156",
        "port": 8080
    },
    "dns": {
        "answers": {
            "name": "Some response string"
        },
        "question": {
            "name": "example.com",
            "type": "Some type"
        }
    },
    "ecs": {
        "version": "8.7.0"
    },
    "elastic_agent": {
        "id": "fc4affb9-ab52-48ec-b9ce-f65f4390f0b9",
        "snapshot": false,
        "version": "8.6.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "network"
        ],
        "dataset": "zscaler_zia.dns",
        "duration": 123456000000,
        "ingested": "2023-02-24T09:42:00Z",
        "kind": "event",
        "type": [
            "info"
        ]
    },
    "input": {
        "type": "tcp"
    },
    "log": {
        "source": {
            "address": "172.29.0.7:53784"
        }
    },
    "network": {
        "protocol": "dns"
    },
    "related": {
        "hosts": [
            "Machine9000"
        ],
        "ip": [
            "89.160.20.112",
            "89.160.20.156"
        ],
        "user": [
            "Owner77"
        ]
    },
    "source": {
        "as": {
            "number": 29518,
            "organization": {
                "name": "Bredband2 AB"
            }
        },
        "geo": {
            "city_name": "Linköping",
            "continent_name": "Europe",
            "country_iso_code": "SE",
            "country_name": "Sweden",
            "location": {
                "lat": 58.4167,
                "lon": 15.6167
            },
            "region_iso_code": "SE-E",
            "region_name": "Östergötland County"
        },
        "ip": "89.160.20.112"
    },
    "tags": [
        "forwarded",
        "zscaler_zia-dns"
    ],
    "user": {
        "email": "some_user@example.com",
        "name": "Owner77"
    },
    "zscaler_zia": {
        "dns": {
            "department": "Unknown",
            "dom": {
                "category": "Professional Services"
            },
            "duration": {
                "milliseconds": 123456
            },
            "hostname": "Machine9000",
            "location": "TestLoc DB",
            "request": {
                "action": "REQ_ALLOW",
                "rule": {
                    "label": "Access Blocked"
                }
            },
            "response": {
                "action": "Some Response Action",
                "rule": {
                    "label": "None"
                }
            }
        }
    }
}

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset.
constant_keyword
event.module
Event module.
constant_keyword
input.type
Type of Filebeat input.
keyword
log.offset
Log offset.
long
log.source.address
Source address from which the log event was read / sent from.
keyword
tags
User defined tags.
keyword
zscaler_zia.dns.department
Department of the user.
keyword
zscaler_zia.dns.dom.category
URL Category of the FQDN in the DNS request.
keyword
zscaler_zia.dns.duration.milliseconds
Duration of the DNS request in milliseconds.
long
zscaler_zia.dns.hostname
N/A
keyword
zscaler_zia.dns.location
Gateway location or sub-location of the source.
keyword
zscaler_zia.dns.request.action
Name of the action that was applied to the DNS request.
keyword
zscaler_zia.dns.request.rule.label
Name of the rule that was applied to the DNS request.
keyword
zscaler_zia.dns.response.action
Name of the action that was applied to the DNS response.
keyword
zscaler_zia.dns.response.rule.label
Name of the rule that was applied to the DNS response.
keyword

firewall

This is the firewall dataset.

Example

An example event for firewall looks as following:

{
    "@timestamp": "2021-12-17T07:27:54.000Z",
    "agent": {
        "ephemeral_id": "83dc07a4-89ea-45b3-b7aa-f72b7e84f8b8",
        "id": "fc4affb9-ab52-48ec-b9ce-f65f4390f0b9",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.6.0"
    },
    "data_stream": {
        "dataset": "zscaler_zia.firewall",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "bytes": 19052,
        "geo": {
            "country_name": "Ireland"
        },
        "ip": "0.0.0.0",
        "port": 443
    },
    "ecs": {
        "version": "8.7.0"
    },
    "elastic_agent": {
        "id": "fc4affb9-ab52-48ec-b9ce-f65f4390f0b9",
        "snapshot": false,
        "version": "8.6.0"
    },
    "event": {
        "action": "drop",
        "agent_id_status": "verified",
        "category": [
            "network"
        ],
        "dataset": "zscaler_zia.firewall",
        "duration": 486000000,
        "ingested": "2023-02-24T09:43:15Z",
        "kind": "event",
        "type": [
            "info"
        ]
    },
    "host": {
        "hostname": "Machine9000"
    },
    "input": {
        "type": "tcp"
    },
    "log": {
        "source": {
            "address": "172.29.0.7:46908"
        }
    },
    "network": {
        "application": "http",
        "community_id": "1:hQwW1HWTOUYlk7y4+T2D+UPDU1c=",
        "protocol": "https",
        "transport": "tcp"
    },
    "related": {
        "hosts": [
            "Machine9000"
        ],
        "ip": [
            "0.0.0.0"
        ],
        "user": [
            "admin77"
        ]
    },
    "rule": {
        "name": [
            "Access Blocked",
            "None"
        ]
    },
    "source": {
        "bytes": 1734,
        "ip": "0.0.0.0",
        "port": 55018
    },
    "tags": [
        "forwarded",
        "zscaler_zia-firewall"
    ],
    "user": {
        "email": "some_user@example.com",
        "name": "admin77"
    },
    "zscaler_zia": {
        "firewall": {
            "aggregate": "No",
            "client": {
                "destination": {
                    "ip": "0.0.0.0",
                    "port": 443
                }
            },
            "department": "Unknown",
            "duration": {
                "avg": 486,
                "milliseconds": 486,
                "seconds": 0
            },
            "ip_category": "Test Name",
            "location": {
                "name": "TestLoc DB"
            },
            "nat": "No",
            "server": {
                "source": {
                    "ip": "0.0.0.0",
                    "port": 0
                }
            },
            "session": {
                "count": 1
            },
            "stateful": "Yes",
            "threat": {
                "category": "None",
                "name": "None"
            },
            "tunnel": {
                "ip": "0.0.0.0",
                "port": 0,
                "type": "ZscalerClientConnector"
            }
        }
    }
}

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset.
constant_keyword
event.module
Event module.
constant_keyword
input.type
Type of Filebeat input.
keyword
log.offset
Log offset.
long
log.source.address
Source address from which the log event was read / sent from.
keyword
tags
User defined tags.
keyword
zscaler_zia.firewall.aggregate
keyword
zscaler_zia.firewall.client.destination.ip
Client destination IP address. For aggregated sessions, this is the client destination IP address of the last session in the aggregate.
ip
zscaler_zia.firewall.client.destination.port
Client destination port. For aggregated sessions, this is the client destination port of the last session in the aggregate.
long
zscaler_zia.firewall.department
Department of the user.
keyword
zscaler_zia.firewall.duration.avg
Average session duration, in milliseconds, if the sessions were aggregated.
long
zscaler_zia.firewall.duration.milliseconds
Session or request duration in milliseconds.
long
zscaler_zia.firewall.duration.seconds
Average session duration, in milliseconds, if the sessions were aggregated.
long
zscaler_zia.firewall.ip_category
URL category that corresponds to the server IP address.
keyword
zscaler_zia.firewall.location.name
Name of the location from which the session was initiated.
keyword
zscaler_zia.firewall.nat
Indicates if the destination NAT policy was applied.
keyword
zscaler_zia.firewall.server.source.ip
Server source IP address. For aggregated sessions, this is the server source IP address of the last session in the aggregate.
ip
zscaler_zia.firewall.server.source.port
Server source port. For aggregated sessions, this is the server source port of the last session in the aggregate.
long
zscaler_zia.firewall.session.count
Number of sessions that were aggregated.
double
zscaler_zia.firewall.stateful
keyword
zscaler_zia.firewall.threat.category
Category of the threat in the Firewall session by the IPS engine.
keyword
zscaler_zia.firewall.threat.name
Name of the threat detected in the Firewall session by the IPS engine.
keyword
zscaler_zia.firewall.tunnel.ip
Tunnel IP address of the client (source). For aggregated sessions, this is the client's tunnel IP address corresponding to the last session in the aggregate.
ip
zscaler_zia.firewall.tunnel.port
Tunnel port on the client side. For aggregated sessions, this is the client's tunnel port corresponding to the last session in the aggregate.
long
zscaler_zia.firewall.tunnel.type
Traffic forwarding method used to send the traffic to the firewall.
keyword

tunnel

This is the tunnel dataset.

Example

An example event for tunnel looks as following:

{
    "@timestamp": "2021-12-30T11:20:12.000Z",
    "agent": {
        "ephemeral_id": "acd73b21-110d-4a61-a22c-0b5a38e01cd4",
        "id": "fc4affb9-ab52-48ec-b9ce-f65f4390f0b9",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.6.0"
    },
    "data_stream": {
        "dataset": "zscaler_zia.tunnel",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "ip": "81.2.69.143"
    },
    "ecs": {
        "version": "8.7.0"
    },
    "elastic_agent": {
        "id": "fc4affb9-ab52-48ec-b9ce-f65f4390f0b9",
        "snapshot": false,
        "version": "8.6.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "network"
        ],
        "dataset": "zscaler_zia.tunnel",
        "id": "1111111111111111111",
        "ingested": "2023-02-24T09:44:32Z",
        "kind": "event",
        "type": [
            "info"
        ]
    },
    "input": {
        "type": "tcp"
    },
    "log": {
        "source": {
            "address": "172.29.0.7:46832"
        }
    },
    "network": {
        "transport": "ipsec ikev 1"
    },
    "related": {
        "ip": [
            "81.2.69.143",
            "81.2.69.145"
        ],
        "user": [
            "81.2.69.145"
        ]
    },
    "source": {
        "ip": "81.2.69.145",
        "port": 0
    },
    "tags": [
        "forwarded",
        "zscaler_zia-tunnel"
    ],
    "user": {
        "name": "81.2.69.145"
    },
    "zscaler_zia": {
        "tunnel": {
            "action": {
                "type": "IPSec Phase2"
            },
            "authentication": {
                "algorithm": "HMAC-SHA-1",
                "type": "None"
            },
            "destination": {
                "end": {
                    "ip": "81.2.69.143"
                },
                "start": {
                    "ip": "81.2.69.143",
                    "port": 0
                }
            },
            "encryption": {
                "algorithm": "AES"
            },
            "ike": {
                "version": 1
            },
            "life": {
                "bytes": 0,
                "time": 3600
            },
            "location": {
                "name": "some-location"
            },
            "policy": {
                "direction": "Inbound SA Policy",
                "protocol": "Any"
            },
            "protocol": "ESP",
            "source": {
                "end": {
                    "ip": "81.2.69.145"
                },
                "start": {
                    "ip": "81.2.69.145",
                    "port": 0
                }
            },
            "spi": "123456789",
            "type": "IPSEC IKEV 1"
        }
    }
}

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset.
constant_keyword
event.module
Event module.
constant_keyword
input.type
Type of Filebeat input.
keyword
log.offset
Log offset.
long
log.source.address
Source address from which the log event was read / sent from.
keyword
tags
User defined tags.
keyword
zscaler_zia.tunnel.action.type
Type of the record. Possible values [ WL_TUNNEL_IPSECPHASE1, WL_TUNNEL_IPSECPHASE2, WL_TUNNEL_EVENT, WL_TUNNEL_SAMPLES ].
keyword
zscaler_zia.tunnel.authentication.algorithm
Authentication algorithm.
keyword
zscaler_zia.tunnel.authentication.type
Authentication type.
keyword
zscaler_zia.tunnel.destination.end.ip
Phase 2 policy proposal - Destination IP end.
ip
zscaler_zia.tunnel.destination.start.ip
Phase 2 policy proposal - Destination IP start.
ip
zscaler_zia.tunnel.destination.start.port
Phase 2 policy proposal - Destination port end.
long
zscaler_zia.tunnel.dpd_packets
Number of DPD packets received in 60-second sample window.
keyword
zscaler_zia.tunnel.encryption.algorithm
Encryption algorithm.
keyword
zscaler_zia.tunnel.ike.version
IKE version (1 or 2).
long
zscaler_zia.tunnel.life.bytes
Life bytes (number of traffic to be transacted through tunnel before renegotiation).
long
zscaler_zia.tunnel.life.time
Lifetime of IKE Phase 1/2 in seconds.
long
zscaler_zia.tunnel.location.name
Location name.
keyword
zscaler_zia.tunnel.policy.direction
N/A
keyword
zscaler_zia.tunnel.policy.protocol
Phase 2 policy proposal - Protocol.
keyword
zscaler_zia.tunnel.protocol
IPSec tunnel protocol type (Zscaler only supports ESP).
keyword
zscaler_zia.tunnel.source.end.ip
Phase 2 policy proposal - Source IP end.
ip
zscaler_zia.tunnel.source.start.ip
Phase 2 policy proposal - Source IP start.
ip
zscaler_zia.tunnel.source.start.port
Phase 2 policy proposal - Source port start.
long
zscaler_zia.tunnel.spi
Security Parameter Index.
keyword
zscaler_zia.tunnel.spi_in
Initiator cookie.
keyword
zscaler_zia.tunnel.spi_out
Responder cookie.
keyword
zscaler_zia.tunnel.type
Tunnel type.
keyword
zscaler_zia.tunnel.vendor.name
Vendor name of the edge device.
keyword

web

This is the web dataset.

Example

An example event for web looks as following:

{
    "@timestamp": "2021-12-31T08:08:08.000Z",
    "agent": {
        "ephemeral_id": "444ca1f4-28b9-45cb-8287-ba44516c521b",
        "id": "08fc14c0-5a92-4649-93f3-68fb5d6c5fbc",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.6.1"
    },
    "data_stream": {
        "dataset": "zscaler_zia.web",
        "namespace": "ep",
        "type": "logs"
    },
    "destination": {
        "ip": "1.128.3.4"
    },
    "ecs": {
        "version": "8.7.0"
    },
    "elastic_agent": {
        "id": "08fc14c0-5a92-4649-93f3-68fb5d6c5fbc",
        "snapshot": false,
        "version": "8.6.1"
    },
    "event": {
        "action": "blocked",
        "agent_id_status": "verified",
        "category": [
            "web"
        ],
        "dataset": "zscaler_zia.web",
        "ingested": "2023-03-15T21:02:55Z",
        "kind": "event",
        "risk_score": 0,
        "type": [
            "info"
        ]
    },
    "http": {
        "request": {
            "bytes": 600,
            "method": "CONNECT",
            "mime_type": "Other",
            "referrer": "None"
        },
        "response": {
            "bytes": 65,
            "status_code": 200
        }
    },
    "input": {
        "type": "http_endpoint"
    },
    "network": {
        "protocol": "http_proxy"
    },
    "related": {
        "hosts": [
            "TestMachine35"
        ],
        "ip": [
            "1.128.3.4"
        ],
        "user": [
            "administrator1",
            "test"
        ]
    },
    "rule": {
        "name": "Zscaler Proxy Traffic",
        "ruleset": "FwFilter"
    },
    "source": {
        "nat": {
            "ip": "1.128.3.4"
        },
        "user": {
            "name": "administrator1"
        }
    },
    "tags": [
        "forwarded",
        "zscaler_zia-web"
    ],
    "url": {
        "domain": "www.example.com",
        "full": "https://www.example.com",
        "original": "https://www.example.com",
        "path": "",
        "scheme": "https"
    },
    "user": {
        "domain": "example.com",
        "email": "test@example.com",
        "name": "test"
    },
    "user_agent": {
        "device": {
            "name": "Other"
        },
        "name": "Other",
        "original": "Windows Microsoft Windows 10 Pro ZTunnel/1.0",
        "os": {
            "full": "Windows 10",
            "name": "Windows",
            "version": "10"
        }
    },
    "zscaler_zia": {
        "web": {
            "app": {
                "class": "General Browsing",
                "name": "General Browsing"
            },
            "ctime": 0,
            "department": "Unknown",
            "device": {
                "hostname": "TestMachine35"
            },
            "dpl": {
                "dictionaries": "None",
                "engine": "None"
            },
            "location": "Test DB",
            "malware": {
                "category": "None"
            },
            "stime": 0,
            "threat": {
                "name": "None"
            },
            "unscannable": {
                "type": "None"
            },
            "url": {
                "category": {
                    "sub": "Web Search",
                    "super": "Information Technology"
                },
                "class": "Business Use"
            }
        }
    }
}

Exported fields

FieldDescriptionType
@timestamp
Event timestamp.
date
data_stream.dataset
Data stream dataset.
constant_keyword
data_stream.namespace
Data stream namespace.
constant_keyword
data_stream.type
Data stream type.
constant_keyword
event.dataset
Event dataset.
constant_keyword
event.module
Event module.
constant_keyword
input.type
Type of Filebeat input.
keyword
log.offset
Log offset.
long
log.source.address
Source address from which the log event was read / sent from.
keyword
tags
User defined tags.
keyword
zscaler_zia.web.app.class
The web application class of the application that was accessed. Equivalent to module.
keyword
zscaler_zia.web.app.name
Cloud application name.
keyword
zscaler_zia.web.bandwidth_throttle
Indicates whether the transaction was throttled due to a configured bandwidth policy.
keyword
zscaler_zia.web.ctime
The time from when the first byte of the request hits the ZEN to the time in which the last byte of the response is sent from the ZEN back to the browser.
long
zscaler_zia.web.department
Department of the user.
keyword
zscaler_zia.web.device.hostname
The obfuscated version of the device owner. This field must be changed manually.
keyword
zscaler_zia.web.dpl.dictionaries
The DLP dictionaries that were matched, if any.
keyword
zscaler_zia.web.dpl.engine
The DLP engine that was matched, if any.
keyword
zscaler_zia.web.encoded_host
Encoded version of the destination host name.
keyword
zscaler_zia.web.file.class
Type of file associated with the transaction.
keyword
zscaler_zia.web.file.type
Type of file associated with the transaction.
keyword
zscaler_zia.web.location
Gateway location or sub-location of the source.
keyword
zscaler_zia.web.malware.category
The category of malware that was detected in the transaction, if any. Also indicates if a file was submitted to the Sandbox engine for analysis and the result of the analysis.
keyword
zscaler_zia.web.malware.class
The class of malware that was detected in the transaction, if any.
keyword
zscaler_zia.web.record.id
N/A
keyword
zscaler_zia.web.stime
The round trip time between the ZEN request and the server.
long
zscaler_zia.web.threat.name
The name of the threat that was detected in the transaction, if any.
keyword
zscaler_zia.web.total.size
Total size, in bytes, of the HTTP transaction; sum of the total request size and total response size.
long
zscaler_zia.web.unscannable.type
Unscannable file type.
keyword
zscaler_zia.web.url.category.sub
Category of the destination URL.
keyword
zscaler_zia.web.url.category.super
Super category of the destination URL.
keyword
zscaler_zia.web.url.class
Class of the destination URL.
keyword

Changelog

VersionDetails
2.8.0
Enhancement View pull request
Update package to ECS 8.7.0.
2.7.3
Enhancement View pull request
Map web login user details to ECS.
2.7.2
Enhancement View pull request
Added categories and/or subcategories.
2.7.1
Enhancement View pull request
Add support for dynamic ECS mapping.

Bug fix View pull request
Resolve the issue related to the user agent field.

Bug fix View pull request
Resolve the issue related to fields that contain a NA value.
2.7.0
Enhancement View pull request
Update package to ECS 8.6.0.
2.6.1
Bug fix View pull request
Remove duplicate fields.
2.6.0
Enhancement View pull request
Update Aggregation visualizations to Lens, Add an on_failure processor to the convert, geo_ip, uri_parts and date processors, remove unnecessary white spaces, mapped to related ecs field and convert double quotes to single quotes.
2.5.0
Enhancement View pull request
Update package to ECS 8.5.0.
2.4.1
Bug fix View pull request
Remap network.protocol to valid values for web data stream.
2.4.0
Enhancement View pull request
Update package to ECS 8.4.0
2.3.1
Enhancement View pull request
Updated the documentation links in README file
2.3.0
Bug fix View pull request
Fix issue related URI parts processor.

Enhancement View pull request
Added support for optionally configuring secret header and secret value for cloud NSS input.
2.2.0
Enhancement View pull request
Update package to ECS 8.3.0.
2.1.0
Enhancement View pull request
Make GA
2.0.0
Enhancement View pull request
Added input for Cloud NSS using HTTP Endpoint input type.
0.2.0
Enhancement View pull request
Update ECS to 8.2
0.1.3
Enhancement View pull request
Updated the image file reference in README file.
0.1.2
Enhancement View pull request
Add documentation for multi-fields.
0.1.1
Enhancement View pull request
Updated the README to describe the Zscaler ZIA setup process in detail.
0.1.0
Enhancement View pull request
Initial draft of the package.