Stream logs to Elasticsearch

VisIQ can forward your governance decision and audit logs into your own Elasticsearch deployment as they happen, indexed as ordinary log documents you can search, alert on and retain under your existing policy. Set it up at Connectors → Log Streaming → Elasticsearch. It takes about three minutes.

Before you start

An Elasticsearch deployment reachable over https from the public internet, and Kibana access to create an API key.
A VisIQ account with the settings:update permission — the log-destination endpoints are guarded by it.

Streaming starts from the moment you save. There is no historical backfill — logs recorded before the destination existed stay in VisIQ’s audit trail and are not replayed into Elastic.

Step 1 — Create a least-privilege API key

In Kibana, go to Stack Management → Security → API keys → Create API key. Turn on Control security privileges and paste this role descriptor:

Elastic role descriptor

{
  "visiq_ingest": {
    "cluster": [
      "monitor"
    ],
    "indices": [
      {
        "names": [
          "logs-*-*"
        ],
        "privileges": [
          "auto_configure",
          "create_doc"
        ]
      }
    ]
  }
}

Then copy the Base64 encoded value — Kibana shows it only once. This is the whole grant, and it is narrow on purpose:

Privilege	What it allows	What it cannot do
`create_doc` on `logs--`	Add new log documents to your `logs-*` data streams.	Update, overwrite, read, or delete anything already in your cluster.
`auto_configure` on `logs--`	Let the data stream create itself on first write.	Touch indices outside the pattern.
`cluster: monitor`	Let the connection check confirm the cluster is reachable.	Any write or admin operation.

If your organisation prefers to provision the key another way, the only requirement is that it can create documents under logs-*-*. A key with more privilege than this works but is not needed.

Step 2 — Point VisIQ at your deployment

Enter your Elasticsearch endpoint URL — the deployment’s base URL, for example https://my-deployment.es.us-central1.gcp.cloud.es.io. Then paste the encoded API key. The endpoint has to clear two checks:

In the browser, a shape check: it must be https, and it must not carry embedded credentials (https://user:pass@host is refused, because such a URL would be stored in plain text and breaks the HTTP client anyway).
On the server, the authoritative one: the host is DNS-resolved and rejected if it lands in a private, reserved or link-local range. This runs at the route boundary and again immediately before every single send, so an endpoint that later resolves inward cannot become an egress path.

A self-hosted cluster therefore has to be reachable at a public address; an endpoint on a private network cannot be used.

Changing the endpoint host requires re-entering your API key. VisIQ refuses to ship a stored credential to a host it was not issued for, so editing the endpoint to a different host locks Save until you paste a fresh key and it verifies against the new host. Changing the path or port of the same host does not trigger this.

Step 3 — Choose what to stream

Pick at least one stream. Both are on by default.

Stream	Contents
Decisions	Action, retrieval, and Human-in-the-Loop governance decision events.
Audit	Platform configuration and access audit log.

You can also set an optional dataset override (for example logs-visiq.decisions-default) if you want the documents to land somewhere other than the default target.

Step 4 — Verify and save

There is no Test connection button. Once the endpoint and key are in, VisIQ writes a test event automatically and Save unlocks when it is verified. If it fails, correct the endpoint or the key and VisIQ re-verifies on its own. Once saved, the connector card shows delivered and failed counts.

How delivery behaves

VisIQ writes through Elasticsearch’s _bulk API with create operations and a deterministic document id per event. That is what makes retries safe: a redelivered document comes back as a 409 version conflict, which VisIQ counts as already-delivered rather than inserting a duplicate. The response handling is deliberate, because _bulk can return 200 while individual documents failed:

Outcome	What VisIQ does
Document accepted, or `409` conflict	Counted delivered.
Per-document `429`	Retried — the batch is not advanced past.
Per-document `400` (mapping or parse error)	Counted as a permanent drop, so one poison document cannot wedge the stream forever.
Whole-request `401` / `403` / `413`	Buffered with back-off, never dropped — a fixable credential or sizing problem must not cost you logs.
Whole-request `429` / `5xx` / network failure	Retried.
`200` with a body VisIQ cannot parse	Retried, not counted delivered. An unparseable `200` usually means a proxy answered instead of Elastic, and VisIQ will not attest delivery it cannot confirm.

The API behind the card

GET needs settings:view; POST needs settings:update.

POST /api/log-destinations

{
  "name": "Elastic — production",
  "type": "elasticsearch",
  "endpoint_url": "https://my-deployment.es.us-central1.gcp.cloud.es.io",
  "streams": ["decisions", "audit"],
  "auth_config": { "api_key": "…" },
  "index_name": "logs-visiq.decisions-default",
  "enabled": true
}

Note that the persisted type is elasticsearch even though the connector card reads “Elasticsearch”. streams must be a non-empty subset of decisions and audit. On a later update, omitting auth_config keeps the stored key unchanged.

Troubleshooting

Verification fails with a 401 or 403

The API key is wrong, or it lacks create_doc on logs-*-*. Re-create it with the role descriptor above and make sure you copied the encoded value, not the key id.

VisIQ says the endpoint is blocked

The host resolved to a private, reserved or link-local address. VisIQ only streams to publicly-resolvable endpoints, and it re-checks before every send rather than trusting the value that was validated at save time.

Save is locked after I edited the endpoint

You changed the host, so a fresh API key is required. This is intentional — a key issued for one cluster is never shipped to another.

Delivered counts look lower than the events I expected

Check the recorded error on the destination. Documents rejected with a mapping or parse error are counted as dropped rather than retried forever; everything else is buffered and retried, so the shortfall is usually temporary.

Stream logs to Datadog

The same streams into Datadog, with the intake host derived from your site.

Stream logs to Rapid7

InsightIDR Custom Logs, where the webhook URL is the whole credential.

Quickstart

Action Governance

Retrieval Governance

Delegation Governance

Unified Rules

Audit Trail

Discovery

Connectors

API Examples

Platform Automation

Partners

Reference

API Reference

Stream logs to Elasticsearch

Before you start

Step 1 — Create a least-privilege API key

Step 2 — Point VisIQ at your deployment

Step 3 — Choose what to stream

Step 4 — Verify and save

How delivery behaves

The API behind the card

Troubleshooting

Stream logs to Datadog

Stream logs to Rapid7

​Before you start

​Step 1 — Create a least-privilege API key

​Step 2 — Point VisIQ at your deployment

​Step 3 — Choose what to stream

​Step 4 — Verify and save

​How delivery behaves

​The API behind the card

​Troubleshooting

​Related

Stream logs to Datadog

Stream logs to Rapid7

Before you start

Step 1 — Create a least-privilege API key

Step 2 — Point VisIQ at your deployment

Step 3 — Choose what to stream

Step 4 — Verify and save

How delivery behaves

The API behind the card

Troubleshooting

Related