Credential Management

Best practices for managing Confluent Cloud credentials, per-cluster API keys, and secrets in different environments.

Overview

LineageBridge needs credentials to access:

Confluent Cloud REST API (environments, clusters, metadata)
Kafka clusters (topics, consumer groups)
Schema Registry (schemas, subjects)
ksqlDB, Flink, Tableflow (queries, statements, mappings)
Data catalogs (Databricks UC, AWS Glue, Google BigQuery)

This guide shows you how to configure, rotate, and secure these credentials.

Understanding Credential Resolution

LineageBridge uses a hierarchical fallback for Kafka cluster credentials:

Per-Cluster Key → Global Kafka Key → Cloud API Key

For each cluster, LineageBridge tries:

Per-cluster credential from CLUSTER_CREDENTIALS JSON map
Global Kafka credential (KAFKA_API_KEY / KAFKA_API_SECRET)
Cloud API key (CONFLUENT_CLOUD_API_KEY / CONFLUENT_CLOUD_API_SECRET)

Example:

# Cloud API key (always required)
LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_KEY=cloud-key-xyz
LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_SECRET=cloud-secret-xyz

# Global Kafka key (used for all clusters except those in CLUSTER_CREDENTIALS)
LINEAGE_BRIDGE_KAFKA_API_KEY=kafka-key-default
LINEAGE_BRIDGE_KAFKA_API_SECRET=kafka-secret-default

# Per-cluster keys (highest priority)
LINEAGE_BRIDGE_CLUSTER_CREDENTIALS={"lkc-prod":{"api_key":"kafka-key-prod","api_secret":"kafka-secret-prod"}}

What happens: - Cluster lkc-prod uses kafka-key-prod (per-cluster) - All other clusters use kafka-key-default (global) - If KAFKA_API_KEY is not set, all clusters use cloud-key-xyz (cloud API fallback)

.env File Structure

LineageBridge loads credentials from a .env file in your project root.

Minimal Setup

Only the cloud API key is required:

# Confluent Cloud (required)
LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_KEY=your-cloud-api-key
LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_SECRET=your-cloud-api-secret

This works if your cloud API key has permissions to read from Kafka clusters, Schema Registry, ksqlDB, Flink, and Tableflow.

Recommended Setup

Use service-scoped credentials for better security:

# ── Confluent Cloud API key (required) ────────────────────────────
LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_KEY=cloud-api-key
LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_SECRET=cloud-api-secret

# ── Global Kafka credentials (optional) ───────────────────────────
LINEAGE_BRIDGE_KAFKA_API_KEY=kafka-api-key
LINEAGE_BRIDGE_KAFKA_API_SECRET=kafka-api-secret

# ── Schema Registry (optional, auto-discovered if unset) ──────────
LINEAGE_BRIDGE_SCHEMA_REGISTRY_ENDPOINT=https://psrc-xxxxx.region.cloud.confluent.cloud
LINEAGE_BRIDGE_SCHEMA_REGISTRY_API_KEY=sr-api-key
LINEAGE_BRIDGE_SCHEMA_REGISTRY_API_SECRET=sr-api-secret

# ── ksqlDB (optional, auto-discovered if unset) ───────────────────
LINEAGE_BRIDGE_KSQLDB_API_KEY=ksql-api-key
LINEAGE_BRIDGE_KSQLDB_API_SECRET=ksql-api-secret

# ── Flink (optional) ──────────────────────────────────────────────
LINEAGE_BRIDGE_FLINK_API_KEY=flink-api-key
LINEAGE_BRIDGE_FLINK_API_SECRET=flink-api-secret

# ── Tableflow (optional) ──────────────────────────────────────────
LINEAGE_BRIDGE_TABLEFLOW_API_KEY=tableflow-api-key
LINEAGE_BRIDGE_TABLEFLOW_API_SECRET=tableflow-api-secret

# ── Databricks Unity Catalog (optional) ───────────────────────────
LINEAGE_BRIDGE_DATABRICKS_WORKSPACE_URL=https://myworkspace.cloud.databricks.com
LINEAGE_BRIDGE_DATABRICKS_TOKEN=dapi...
LINEAGE_BRIDGE_DATABRICKS_WAREHOUSE_ID=abc123

# ── AWS Glue (optional, uses AWS credentials from environment) ────
LINEAGE_BRIDGE_AWS_REGION=us-east-1

# ── Google Cloud (optional) ───────────────────────────────────────
LINEAGE_BRIDGE_GCP_PROJECT_ID=my-project
LINEAGE_BRIDGE_GCP_LOCATION=us

Per-Cluster Credentials

For production setups with multiple clusters, use CLUSTER_CREDENTIALS:

LINEAGE_BRIDGE_CLUSTER_CREDENTIALS='{"lkc-prod-us":{"api_key":"prod-us-key","api_secret":"prod-us-secret"},"lkc-prod-eu":{"api_key":"prod-eu-key","api_secret":"prod-eu-secret"},"lkc-staging":{"api_key":"staging-key","api_secret":"staging-secret"}}'

Formatting tips:

Use single quotes around the JSON to avoid shell escaping issues
Minify the JSON (no extra whitespace)
Each cluster ID maps to an object with api_key and api_secret fields

JSON structure:

{
  "lkc-prod-us": {
    "api_key": "prod-us-key",
    "api_secret": "prod-us-secret"
  },
  "lkc-prod-eu": {
    "api_key": "prod-eu-key",
    "api_secret": "prod-eu-secret"
  }
}

Auto-Provisioning with Confluent CLI

LineageBridge includes a script to auto-provision a cloud API key using the Confluent CLI.

Prerequisites

Install the Confluent CLI:

$ brew install confluentinc/tap/cli
# or: https://docs.confluent.io/confluent-cli/current/install.html

Log in to Confluent Cloud:
```
$ confluent login
```

Run the Script

$ bash scripts/ensure-cloud-key.sh

What it does:

Checks if .env contains CONFLUENT_CLOUD_API_KEY
If missing, creates a cloud API key via confluent api-key create --resource cloud
Appends the key to .env with a timestamp comment
Exits silently if a key already exists

Expected output:

  ┌──────────────────────────────────────────────────────────┐
  │  No Confluent Cloud API key found.                       │
  │  Creating one via the Confluent CLI...                   │
  └──────────────────────────────────────────────────────────┘

  Creating Cloud API key...

  ✓ Cloud API key created: ABCD...WXYZ
  ✓ Saved to .env

Use in Scripts

The make ui target runs this script automatically:

$ make ui
# Auto-provisions cloud key if missing, then starts the UI

For headless automation, run the script before extraction:

#!/bin/bash
set -euo pipefail

bash scripts/ensure-cloud-key.sh
uv run lineage-bridge-extract --env env-abc123

Encrypted Cache

LineageBridge stores UI state (selected environments, per-cluster credentials) in an encrypted local cache.

Cache Location

~/.lineage_bridge/cache.json        # Encrypted cache
~/.lineage_bridge/.cache_key        # Fernet encryption key (owner-only)

What's Cached

Selected environments and clusters (plaintext)
Per-cluster API credentials entered via UI (encrypted)
Schema Registry credentials (encrypted)
Flink credentials (encrypted)
Auto-provisioned API keys (encrypted)

Encryption

LineageBridge uses Fernet symmetric encryption (AES-128-CBC with HMAC-SHA256):

On first run, generates a Fernet key and stores it in ~/.lineage_bridge/.cache_key
Sets file permissions to 600 (owner-only read/write)
Encrypts sensitive fields before writing to cache.json
Decrypts on load

Security properties:

Key never leaves the local machine
Cache file is useless without the key file
Key file has restricted permissions (owner-only)

Cache Invalidation

To clear the cache:

$ rm -rf ~/.lineage_bridge/

Or delete specific entries by editing cache.json (decrypt it first, or just delete the file).

Credential Rotation

Cloud API Key Rotation

Create a new cloud API key:

$ confluent api-key create --resource cloud --description "LineageBridge (new)"

Update .env:

LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_KEY=new-cloud-key
LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_SECRET=new-cloud-secret

Verify it works:
```
$ uv run lineage-bridge-extract
```

Delete the old key:

$ confluent api-key delete old-cloud-key --force

Cluster-Scoped Key Rotation

Create a new cluster-scoped API key:

$ confluent api-key create --resource lkc-abc123 --description "LineageBridge (new)"

Update .env (per-cluster credentials):

# Old:
LINEAGE_BRIDGE_CLUSTER_CREDENTIALS='{"lkc-abc123":{"api_key":"old-key","api_secret":"old-secret"}}'

# New:
LINEAGE_BRIDGE_CLUSTER_CREDENTIALS='{"lkc-abc123":{"api_key":"new-key","api_secret":"new-secret"}}'

Verify it works:

$ uv run lineage-bridge-extract --env env-xyz --cluster lkc-abc123

Delete the old key:

$ confluent api-key delete old-key --force

Databricks Token Rotation

Generate a new personal access token in the Databricks workspace UI (User Settings → Access Tokens)

Update .env:

LINEAGE_BRIDGE_DATABRICKS_TOKEN=new-token

Verify it works:

$ uv run lineage-bridge-extract --env env-xyz
# Check that UC enrichment succeeds

Revoke the old token in the Databricks UI

AWS Credentials Rotation

LineageBridge uses the standard AWS credential chain (environment variables, ~/.aws/credentials, IAM role).

To rotate:

Update AWS credentials via aws configure or environment variables

Verify it works:

$ aws glue get-databases --region us-east-1
$ uv run lineage-bridge-extract --env env-xyz

No changes to .env needed unless you're setting AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY explicitly.

Secrets in CI/CD

GitHub Actions

Store credentials as repository secrets:

Go to your repository → Settings → Secrets and variables → Actions
Add secrets:
LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_KEY
LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_SECRET
LINEAGE_BRIDGE_DATABRICKS_WORKSPACE_URL
LINEAGE_BRIDGE_DATABRICKS_TOKEN
(others as needed)

Reference them in your workflow:

env:
  LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_KEY: ${{ secrets.LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_KEY }}
  LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_SECRET: ${{ secrets.LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_SECRET }}
  LINEAGE_BRIDGE_DATABRICKS_WORKSPACE_URL: ${{ secrets.LINEAGE_BRIDGE_DATABRICKS_WORKSPACE_URL }}
  LINEAGE_BRIDGE_DATABRICKS_TOKEN: ${{ secrets.LINEAGE_BRIDGE_DATABRICKS_TOKEN }}

Do NOT commit credentials to .env in CI. Use secrets or environment variables.

GitLab CI

Store credentials as CI/CD variables:

Go to your project → Settings → CI/CD → Variables
Add variables (mark as "Masked" and "Protected"):
LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_KEY
LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_SECRET

Reference them in .gitlab-ci.yml:

extract-lineage:
  script:
    - export LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_KEY="${LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_KEY}"
    - export LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_SECRET="${LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_SECRET}"
    - uv run lineage-bridge-extract

Docker Secrets

For Docker deployments, use Docker secrets or environment variables:

$ docker run \
  -e LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_KEY="$CONFLUENT_KEY" \
  -e LINEAGE_BRIDGE_CONFLUENT_CLOUD_API_SECRET="$CONFLUENT_SECRET" \
  lineage-bridge:latest \
  lineage-bridge-extract

Or mount a .env file as a volume (be careful with file permissions):

$ docker run \
  -v "$(pwd)/.env:/app/.env:ro" \
  lineage-bridge:latest \
  lineage-bridge-extract

Security Best Practices

1. Use Service-Scoped API Keys

Prefer cluster-scoped, Schema Registry-scoped, and ksqlDB-scoped keys over cloud API keys.

Why: Cloud API keys have broad permissions. Service-scoped keys limit the blast radius if compromised.

How:

# Create cluster-scoped key
$ confluent api-key create --resource lkc-abc123 --description "LineageBridge read-only"

# Create Schema Registry key
$ confluent api-key create --resource lsrc-xyz --description "LineageBridge SR read"

2. Restrict File Permissions

.env should be readable only by you:

$ chmod 600 .env

The cache directory is auto-secured by LineageBridge (permissions 700 on the directory, 600 on files).

3. Rotate Credentials Regularly

Rotate API keys every 90 days:

Create new keys
Update .env and CI/CD secrets
Verify everything works
Delete old keys

4. Use Read-Only Keys

LineageBridge only reads metadata. It never writes to Kafka, Schema Registry, or ksqlDB (except when pushing lineage to catalogs, which is opt-in).

Grant minimal permissions: - Kafka: DeveloperRead or ResourceOwner (read-only) - Schema Registry: ResourceOwner (read schemas) - ksqlDB: read-only access (no CREATE, DROP, INSERT)

5. Never Commit `.env` to Git

Add .env to .gitignore:

$ echo ".env" >> .gitignore

Use .env.example as a template (no real credentials).

6. Audit Access

Check which API keys exist:

$ confluent api-key list

Delete unused keys:

$ confluent api-key delete <key-id> --force

Troubleshooting

"401 Unauthorized"

Cause: The API key is invalid or lacks required permissions.

Solution:

Verify the key exists and is active:
```
$ confluent api-key list
```

Check which credential is being used. LineageBridge logs the credential resolution order:

[INFO] Using per-cluster credential for lkc-abc123
[INFO] Using global Kafka credential for lkc-def456
[INFO] Using cloud API key for lkc-ghi789

Test the key manually:

$ confluent kafka topic list --cluster lkc-abc123 \
    --api-key <key> --api-secret <secret>

If the key is wrong, update .env and restart.

"403 Forbidden"

Cause: The API key is valid but lacks the required permissions.

Solution:

Check the key's role:
```
$ confluent api-key describe <key>
```
Grant read permissions via the Confluent Cloud UI (IAM → API Keys).

For cluster-scoped keys, ensure they have ACLs:

$ confluent kafka acl list --cluster lkc-abc123 --api-key <key> --api-secret <secret>

"Could not find .env file"

Cause: .env doesn't exist in your project root.

Solution:

Copy the example:
```
$ cp .env.example .env
```
Fill in your credentials.
Verify the file exists:
```
$ ls -la .env
```

"Cluster credential JSON is invalid"

Cause: Malformed CLUSTER_CREDENTIALS JSON.

Solution:

Validate the JSON:

$ echo "$LINEAGE_BRIDGE_CLUSTER_CREDENTIALS" | jq .

Check for common issues:
Missing quotes around keys or values
Trailing commas
Single quotes inside double-quoted strings (use \' or switch outer quotes)

Use a JSON formatter:

$ echo '{"lkc-abc":{"api_key":"foo","api_secret":"bar"}}' | jq -c .

Next Steps

Set up multi-environment extraction with per-cluster credentials
Deploy with Docker to isolate secrets in containers
Integrate with CI/CD to manage secrets in GitHub Actions or GitLab CI

Credential Management

Overview

Understanding Credential Resolution

.env File Structure

Minimal Setup

Recommended Setup

Per-Cluster Credentials

Auto-Provisioning with Confluent CLI

Prerequisites

Run the Script

Use in Scripts

Encrypted Cache

Cache Location

What's Cached

Encryption

Cache Invalidation

Credential Rotation

Cloud API Key Rotation

Cluster-Scoped Key Rotation

Databricks Token Rotation

AWS Credentials Rotation

Secrets in CI/CD

GitHub Actions

GitLab CI

Docker Secrets

Security Best Practices

1. Use Service-Scoped API Keys

2. Restrict File Permissions

3. Rotate Credentials Regularly

4. Use Read-Only Keys

5. Never Commit .env to Git

6. Audit Access

Troubleshooting

"401 Unauthorized"

"403 Forbidden"

"Could not find .env file"

"Cluster credential JSON is invalid"

Next Steps

5. Never Commit `.env` to Git