Exporting security events

© Pi Soft, 2018-2026 · Tricryption Engine 8.1

A reference implementation for exporting the Tricryption Engine's security events to a SIEM. Three producers write one versioned envelope:

producer	what it records	how it leaves
key-server	every audit row the key server inserts into te_auditlogex	change data capture on the key database (a reference pipeline for it is the evaluation stack's SIEM overlay, below) -- the key server itself is unchanged
gateway	every refusal the REST Gateway answers: 401, 403, 429, 503, and a 400 with a PROOF_* code	an append-only outbox the Gateway writes before it answers, when its security-event capture is on (TE_GATEWAY_SECURITY_EVENTS_DIR)
workflow	the workflow adapter's grant, cleanup, scheduled and observed expiry outcomes, and its own refusals	an append-only outbox the adapter commits before it returns (the Python starter's workflow/export_outbox.py)

dispatcher/dispatch.mjs drains either outbox to a durable JSON file or to Kafka. schema/ holds the envelope (security-event.schema.json) and a standard-library validator (validate.py).

The envelope, version 1

additionalProperties: false at every level: the fields below are an allow-list. No detail, hidden link, t-Tag, jsonStr/attribute map, plaintext, ciphertext, key material, credential, bearer or DPoP proof is ever a field, and an event carrying one is refused by every producer and by the dispatcher.

field	type	set by	value
schema_version	integer, const 1	all	the envelope version
event_id	string	all	see "Event ids" below
producer	key-server | gateway | workflow	all
deployment	string	all	key server: the pipeline's configured deployment name; Gateway: TE_GATEWAY_SECURITY_EVENTS_DEPLOYMENT, else the key server's system id once the boot probe has read it (unknown before; dispatch.mjs --deployment fills an unknown); workflow: security_events.deployment in the starter's config, else unknown
generation	string	all	see "Generations" below
event_type	enum	all	ks.audit, gateway.refusal, workflow.grant, workflow.cleanup, workflow.expiry_scheduled, workflow.expiry_observed, workflow.refusal, producer.audit_degraded
occurred_at	RFC 3339 UTC	all	key server: te_ltime (epoch seconds); Gateway and workflow: the moment the outcome was decided
ingested_at	RFC 3339 UTC	the pipeline	absent until the pipeline (Logstash) sets it
scheduled_for	RFC 3339 UTC	workflow	workflow.expiry_scheduled only: the grant's endTime
operation	string	all	key server: te_opid by the operation name the Console's audit view shows; Gateway: METHOD /route-pattern (ids elided); workflow: the adapter operation (grant_step, decrypt_fragment, close_step, expiry)
outcome	succeeded | refused | failed | scheduled | observed	all	key server: te_status 0 -> succeeded; otherwise refused when te_reason is aclDenied or roleDenied (an authorization refusal), else failed
reason	string	all	key server: te_reason by its stable KEY, the one the Console's audit view filters on (e.g. none, aclDenied; a code the reference pipeline's table lacks keeps its number as a string); Gateway: the refusal's code (a block refusal's exported reason is AUDIT_CAPTURE_UNAVAILABLE although its wire code is CAPACITY_UNAVAILABLE); workflow: the WF_* code, the key server's code, or ACL_ENTRY_ADDED/ACL_ENTRY_UPDATED/END_TIME/REVOKED/ALREADY_GONE/DEADLINE_PASSED
actor	{kind: known|unavailable, system?, principal?, name?}	all	key server: te_prs (system), te_prid (principal), as strings; Gateway: the session's user (name) when the session guard resolved one; workflow: the acting principal (the workflow's owner, or the step's actor on a decrypt)
object	{kind: known|unavailable, system?, principal?, object_id?, engine_id?}	all	key server: te_objprs, te_objprid, te_objid, te_engineid, as strings; Gateway: unavailable; workflow: the fragment's opaque id (object_id) and the recipient principal (system, principal)
correlation	{label: exact|association|unavailable, request_id?, workflow_id?, step_id?, run_id?}	all	see "Correlation" below
source	{table, row_id} strings	key server only	te_auditlogex, te_rowid
http	{status, method, route}	gateway.refusal only	the answered status, the method, the route pattern
degraded	{failures, last_error}	producer.audit_degraded only	the producer's failed-append counter and the last error's CLASS (never a message or a path)

The key-server columns NOT exported, deliberately: te_objdet (free-form detail), te_hlink (the hidden link), and the row's JSON/attribute map.

Event ids

• Key server: <deployment>:<generation>:te_auditlogex:<te_rowid>, composed by the pipeline from the row -- so the initial snapshot and the change stream produce the SAME id for the same row, and a destination that keys documents by event_id deduplicates them.
• Gateway and workflow: a UUID v4 minted ONCE, at append, and persisted with the row. A re-send (a dispatcher that crashed between the destination's acknowledgement and its mark) carries the same id.

Generations

An outbox mints its generation (a UUID, in its meta table) when the file is CREATED. A deleted or re-initialised outbox, or a key database restored from backup, is a NEW generation; an existing generation keeps its ids. For the key server the pipeline is configured with its generation, and an operator restoring the database changes it.

Correlation

• exact -- within ONE producer only. A Gateway refusal is the event itself (label: exact); it has no key-server row by construction, because a refused request never reached the key server. A workflow outcome is its own row and names workflow_id, step_id and the adapter process's run_id.
• association -- actor + object + a time window across producers. Set by an analyst's query, never by a producer. Time proximity alone proves nothing.
• unavailable -- otherwise (every key-server row: the audit row carries no request id).

Expiry

workflow.expiry_scheduled is written when a grant with an endTime is made (every adapter grant has one); scheduled_for is that endTime. workflow.expiry_observed is written, once per grant, when the adapter sees the window enforced: a step found past its deadline (retry_pending, and so reconcile on a restart), or a decrypt the key server refused after endTime. A key-server ks.audit row of the refused decrypt is the key server's own observation.

Delivery

At least once. Each producer's row is durable before the outcome is visible; the dispatcher marks a batch exported only after the sink acknowledged it (the file fsynced, or Kafka's acks: -1), so a crash between the two re-sends that batch on the next run and the destination deduplicates by event_id. An event the envelope refuses is reported by the dispatcher and left pending, never sent.

Your own PostgreSQL

On an installed system the key database is the customer's own PostgreSQL (Amazon RDS, Cloud SQL, a server of their own), not the Compose stack's. Its owner prepares the capture once; the export then reads it exactly as it reads the stack's.

1. Logical decoding on. wal_level=logical. On Amazon RDS that is the static parameter rds.logical_replication=1 in the instance's parameter group, in force after a reboot (or from the creation of an instance made from that group).

2. The capture role and the publication, as the database's administrator, with the audit table's owner's rights for the publication (SET ROLE <owner> where the administrator is a member of it):

   CREATE ROLE te_siem_cdc LOGIN PASSWORD '<a password of your own>';
   GRANT rds_replication TO te_siem_cdc;          -- Amazon RDS; elsewhere: ALTER ROLE te_siem_cdc REPLICATION
   SET ROLE <the audit table's owner>;
   REVOKE ALL ON ALL TABLES IN SCHEMA public FROM te_siem_cdc;
   GRANT CONNECT ON DATABASE <key database> TO te_siem_cdc;
   GRANT USAGE ON SCHEMA public TO te_siem_cdc;
   GRANT SELECT (te_rowid, te_ltime, te_opid, te_status, te_reason, te_prs, te_prid, te_objprs, te_objprid, te_objid, te_engineid) ON te_auditlogex TO te_siem_cdc;
   CREATE PUBLICATION te_siem_audit FOR TABLE te_auditlogex (te_rowid, te_ltime, te_opid, te_status, te_reason, te_prs, te_prid, te_objprs, te_objprid, te_objid, te_engineid);

   The role reads those eleven columns and nothing else, and needs no other grant.

3. The Gateway's capture: tricryption-tier-provision --security-events-dir /var/lib/pisoft/console-gateway/security-events (it writes TE_GATEWAY_SECURITY_EVENTS_DIR into the Gateway's environment and makes the directory for the service user; a rerun without the option turns capture off). For the reference overlay below add --security-events-deployment tricryption-eval: the overlay is one deployment, its ingest role writes te-security-events-tricryption-eval-* only, and it labels the key server's rows tricryption-eval too -- an event under any other name is refused (403), retried for ever and never indexed, which the overlay's own check reports. Drain that outbox ON the Gateway's machine with the dispatcher the release ships: /opt/pisoft/tricryption/node/bin/node /opt/pisoft/tricryption/console-gateway/integrations/security-events/dispatcher/dispatch.mjs --outbox /var/lib/pisoft/console-gateway/security-events/security-events.db --producer gateway --sink kafka --brokers <your brokers> --topic <topic> --follow (as root: the outbox is the service user's, 700).

4. The reference overlay against it (an evaluation of the pipeline, not a production topology). The evaluation stack's SIEM overlay -- a Compose overlay with Kafka Connect, Logstash and OpenSearch, which comes with the evaluation stack, not with this bundle -- can read your database instead of the stack's. Hand it a directory holding keydb.env (TE_SIEM_KEYDB_HOST=<the server's name>, TE_SIEM_KEYDB_PORT=<port>, TE_SIEM_KEYDB_NAME=<key database>), ca.pem (the server's CA bundle) and cdc-password, set TE_SIEM_KEYDB=<dir> and the evaluation image tag, and start it. Nothing of the Compose stack starts; the overlay's one-shot tricryption-eval-siem-keydb-init reads the directory read-only, checks wal_level, the publication's column list and the slots AS te_siem_cdc, and creates nothing in your database. The server's name resolves to the Docker host (so a forward there reaches it), and verify-full checks that name against the server's own certificate. The overlay's check grades it as it grades the stack's.

Measured with an Amazon RDS for PostgreSQL 16 key database reached over verify-full: the database prepared by steps 1 and 2, the Gateway's capture by step 3, and the overlay reading the database by step 4: a refused Gateway request and an administrator login were each found by event_id in the deployment's index, and the overlay's check passed with no index write denied.

Removing the integration

Nothing cryptographic depends on it:

1. Unset TE_GATEWAY_SECURITY_EVENTS_DIR and restart the Gateway (capture is off; nothing is created), and pass export_outbox=None to WorkflowAdapter; stop the dispatchers.
2. Delete the outboxes (security-events.db and its -wal/-shm files).
3. On the key database: drop every te_siem_* replication slot (pg_drop_replication_slot; an unread slot keeps WAL for ever), then DROP PUBLICATION te_siem_audit, DROP OWNED BY te_siem_cdc and DROP ROLE te_siem_cdc.

The key server, the Gateway's answers and the adapter's grants behave exactly as before. On the Compose evaluation stack, the reference overlay's own remove does all three (its containers and tricryption-eval-siem-* volumes too); measured: the stack's encrypt/decrypt roundtrip passes afterwards.