Qlik Data Gateway Monitor

The Qlik Data Gateway Monitor, a.k.a. the ConnectorAgent Monitor, is a tool developed and maintained by Qlik Customer First Data Analytics (CF QDA).

Features

Monitors the Qlik Data Gateway - Direct Access Windows service process and restarts it if necessary ✅
Saves status endpoint responses for diagnostic purposes ✅
Provides an API for collecting all Direct Access Agent service logs ✅
- DirectAccessAgent (DirectAccessAgent.log and DirectAccessAgent_YYYYMMDDHHmmss.log)
- ConnectorAgent (connector-agent_logs_YYMMDD.txt)
- ODBC Connector (odbc-connector_logs.txt and odbc-connector_logs_[data_port].txt)
- SAP BW (QvSAPBwConnector-YYYYMMDD_HHmmssfff-[data_port].log)
- SAP SQL (QvSAPSqlConnector-YYYYMMDD_HHmmssfff-[data_port].log)
- SAP Package / ODP (QvSapConnectorPackage-[data_port].log)
- File Connector (file-connector_logs_YYMMDD.txt)
- REST Connector (rest-connector_logs.txt)
Collects Windows Application and System event logs covering the specified time period ✅
Optionally schedules automatic collection and upload of logs to FileCloud ✅
Transform logs as needed to streamline their ingest into Qlik Sense ⏳
Being considered for future releases
- Save server health metrics
- Include copy of current agent configuration.properties file in log collection
- Web proxy support for automatic log upload to FileCloud
  - Please note: This feature is not available in the current version of the Monitor.
  - If you need this feature, or have another idea you would like to share regarding the Monitor, then please log it here and we will consider it for future releases.

System Requirements

Same as the Qlik Data Gateway - Direct Access 1.6.8 and above:
- Windows Server 2016 or later
- .NET 8.0.x Runtime (x64) and ASP.NET Core Runtime 8.0.x (x64) (latest patch)
- Setting up Qlik Data Gateway - Direct Access > System prerequisites

Installing

Latest Installer - Qlik.ConnectorAgent.Monitor.Installer-0.5.10.exe
The current version is 0.5.10 which includes optional automatic upload to FileCloud. The Monitor can be installed on an active gateway without causing interruptions. Just download and run the installer. No restarts or manual Windows service starts are required.

What's Changed in the Latest Release

Resolves an issue where the Monitor service is unable to start after installing Direct Access Gateway 1.7.4
- Note: In-place upgrades might be impacted by the issue that this release fixes. If that happens, uninstall the Data Gateway Monitor and install the latest version. Previous settings will not be lost.

How It Works

The Monitor sends a request to the ConnectorAgent's status endpoint which triggers the agent to check itself and all of its registered connectors at the time of request.
The JSON payload of each response is logged to a C:\ProgramData\Qlik\Gateway\Monitor\Logs\connector-agent_status_YYYYMMDD.log file.
The statusCheckResult in that payload must be PASS.
The Monitor also waits for the ConnectorAgent.exe process to be running and that it returns at least one PASS (after the Monitor starts up) before any non-PASS is counted as a failure.
After the first failure, it checks for a PASS every minute (frequency is configurable via StatusMonitor.CheckInterval) for AGENT_HEALTH_FAIL_MINUTES_LIMIT + StatusMonitor.RestartThreshold minutes (5 + 2.5 = 7.5 minutes by default, also configurable) after which it kills the ConnectorAgent.exe process and attempts a restart of its associated Windows service.
The AGENT_HEALTH_FAIL_MINUTES_LIMIT value is read from the C:\ProgramData\Qlik\Gateway\configuration.properties file and the StatusMonitor settings are in, or can be added to the C:\ProgramData\Qlik\Gateway\Monitor\appsettings.json file.

The configuration defaults work for most use cases. A default appsettings.json file is installed along with the tool. An environment-specific appsettings.json file can be created in the C:\ProgramData\Qlik\Gateway\Monitor folder. Some features, like automatic log upload, require either creating or merging settings into this file as shown in the next section. Changing any Monitor settings, other than the LogUploader settings, currently require restarting its Windows service.

Manual Log Collection

The Monitor can be used directly on the gateway server to collect logs and save them to a local folder via its API, or a web browser using the steps below.

Open a modern browser, such as Chrome or Edge (Microsoft Edge comes preinstalled on recent server versions), and enter the following URL: http://localhost:5051/logs/download?start=YYYY-MM-DD[THH:mm:ss][+-HH:mm][&end=YYYY-MM-DD[THH:mm:ss][+-HH:mm]] where:
- YYYY-MM-DD is the date from which you want to start collecting logs.
- THH:mm:ss is the letter T followed by the time of day in 24-hour format from which you want to start collecting logs. This is optional.
- +-HH:mm is the time zone offset (relative to UTC) for the data and/or time provided. If a time zone is not specified, then it is assumed to be UTC. This is optional.
- It is recommended to specify the start date at least two (2) days before the period of interest to ensure all pertinent logs are included.
- An optional end date can be specified in the same format as the start date. If not specified, then the current date and time plus 5 minutes is used.
The browser should either save the logs as zip file to your Downloads folder, or prompt you to save the logs to a location of your choosing.
- Depending on the length of time specified, the zip file may be large and take a while to start downloading.
- The Direct Access gateway and Data Gateway Monitor both rotate component logs on a regular basis as needed.

LogUploader

The automatic log collection and upload to FileCloud feature is designed to ensure continuous log coverage is available to Qlik Product Support and streamline diagnostics during a case. To enable the feature, add a LogUploader subsection under the Tasks section in the C:\ProgramData\Qlik\Gateway\Monitor\appsettings.json file. Create the file and sections as needed. Example shown below.
Schedule is a standard cron expression. The example below, 0 9 * * *, means collect logs every day at 9 AM. Use 24-hour time representation, ex. 3PM server time would be 15.
UploadUrl is the FileCloud upload share URL that we provide to customers when a SalesForce case is opened. Please note the following restrictions:
- Only FileCloud links starting with https://files.qlik.com/url/ are supported.
- If you set this URL to a manually-created, private shared folder, then it must have Upload-only permissions.
- Other configurations and upload targets, ex. SharePoint, a local folder or fileshare, are not supported.
Days controls how many days the Monitor will include if it cannot find a recent successful upload in the C:\ProgramData\Qlik\Gateway\Monitor\tasks.db database. If the Days setting is not included in the appsettings.json file, then its value defaults to 2.
- For example, if the Monitor runs at 09:05 local-server-time on 2024-01-03 with the setting "Days": 2, then it would include files anf log events created or modified between 2024-01-01 00:00 and 2024-01-01 09:10.
Settings can be changed by editing and saving the file with any text editor. Restarting the service is not required.
To confirm that settings have been applied, open the latest C:\ProgramData\Qlik\Gateway\Monitor\Logs\monitor_YYYYMMDD.log file and look for an entry like: [INF] (Qlik.ConnectorAgent.Monitor) Logs will be automically uploaded to https://files.qlik.com/url/-redacted- at 03:00 PM

{
    "Tasks": {
        "LogUploader": {
            "Enabled": true,
            "UploadUrl": "https://files.qlik.com/url/-replace-me-",
            "Schedule": "5 9 * * *"
        }
    }
}

LogUploader settings can be changed at any time via a text editor and those settings will be applied on-the-fly (without restarting its Windows service).

License / Disclaimer

This tool is provided "AS IS", without any warranty, under the MIT License - see the LICENSE file for details.
It is not an official Qlik product, but issues and proposed enhancements can be reported here for potential consideration.

Qlik Data Gateway Monitor

Features

System Requirements

Installing

What's Changed in the Latest Release

How It Works

Configuration

Manual Log Collection

LogUploader

License / Disclaimer

On this page