> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-mintlify-1d264819.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring Temporal Cloud metrics with ClickStack

> Monitoring Temporal Cloud Metrics with ClickStack

export const TrackedLink = ({href, eventName, children, ...rest}) => {
  const handleClick = () => {
    try {
      if (typeof window !== "undefined" && window.galaxy && eventName) {
        window.galaxy.track(eventName, {
          interaction: "click"
        });
      }
    } catch (e) {}
  };
  return <a href={href} onClick={handleClick} {...rest}>
      {children}
    </a>;
};

export const Image = ({img, alt, size}) => {
  return <Frame>
      <img src={img} alt={alt} />
    </Frame>;
};

<Info>
  **Warning**

  OpenMetrics support in the Temporal platform is available in [Public Preview](https://docs.temporal.io/evaluate/development-production-features/release-stages#public-preview). Refer to [their documentation](https://docs.temporal.io/cloud/metrics/openmetrics) for more information.
</Info>

Temporal offers an abstraction for building simple, sophisticated, resilient applications.

<Info>
  **TL;DR**

  Monitor Temporal Cloud metrics in ClickStack using the OTel Prometheus receiver. Includes a pre-built dashboard.
</Info>

<h2 id="existing-temporal">
  Integration with existing Temporal Cloud
</h2>

This section covers configuring ClickStack by configuring the ClickStack OTel collector with the Prometheus receiver.

<h2 id="prerequisites">
  Prerequisites
</h2>

* ClickStack instance running
* Existing Temporal Cloud account
* HTTP network access from ClickStack to your Temporal Cloud

<Steps>
  <Step>
    <h4 id="create-temporal-cloud-key">
      Create Temporal Cloud key
    </h4>

    Ensure you have a Temporal Cloud API key. This can be created by following the [Authentication guide](https://docs.temporal.io/production-deployment/cloud/metrics/openmetrics/api-reference#authentication) in the Temporal documentation.

    <Warning>
      **Key file**

      Ensure these credentials are stored in a file `temporal.key` in the same directory as the config file created below. This key should just be stored as text with no preceding or following spaces.
    </Warning>
  </Step>

  <Step>
    <h4 id="custom-otel">
      Create custom OTel collector configuration
    </h4>

    ClickStack allows you to extend the base OpenTelemetry collector configuration by mounting a custom configuration file and setting an environment variable. The custom configuration is merged with the base configuration managed by HyperDX via OpAMP.

    Create a file named `temporal-metrics.yaml` with the following configuration:

    ```yaml title="temporal-metrics.yaml" theme={null}
    receivers:
      prometheus/temporal:
        config:
          scrape_configs:
          - job_name: 'temporal-cloud'
            scrape_interval: 60s
            scrape_timeout: 30s
            honor_timestamps: true
            scheme: https
            authorization:
              type: Bearer
              credentials_file: /etc/otelcol-contrib/temporal.key
            static_configs:
              - targets: ['metrics.temporal.io']
            metrics_path: '/v1/metrics'

    processors:
      resource:
        attributes:
          - key: service.name
            value: "temporal"
            action: upsert

    service:
      pipelines:
        metrics/temporal:
          receivers: [prometheus/temporal]
          processors:
            - resource
            - memory_limiter
            - batch
          exporters:
            - clickhouse
    ```

    This configuration:

    * Connects to Temporal Cloud at `metrics.temporal.io`
    * Collects metrics every 60 seconds
    * Collects [key performance metrics](https://docs.temporal.io/production-deployment/cloud/metrics/openmetrics/metrics-reference)
    * **Sets the required `service.name` resource attribute** per [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/resource/#service)
    * Routes metrics to the ClickHouse exporter via a dedicated pipeline

    <Note>
      - You only define new receivers, processors, and pipelines in the custom config
      - The `memory_limiter` and `batch` processors and `clickhouse` exporter are already defined in the base ClickStack configuration - you just reference them by name
      - The `resource` processor sets the required `service.name` attribute per OpenTelemetry semantic conventions
      - For multiple Temporal cloud accounts, customize `service.name` to distinguish them (e.g., `"temporal-prod"`, `"temporal-dev"`)
    </Note>
  </Step>

  <Step>
    <h4 id="load-custom">
      Configure ClickStack to load custom configuration
    </h4>

    To enable custom collector configuration in your existing ClickStack deployment, you must:

    1. Mount the custom config file at `/etc/otelcol-contrib/custom.config.yaml`
    2. Set the environment variable `CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml`
    3. Mount the `temporal.key` file at `/etc/otelcol-contrib/temporal.key`
    4. Ensure network connectivity between ClickStack and Temporal

    All commands assume they're executed from the sample directory as where `temporal-metrics.yaml` and `temporal.key` are stored.

    <h5 id="docker-compose">
      Option 1: Docker Compose
    </h5>

    Update your ClickStack deployment configuration:

    ```yaml theme={null}
    services:
      clickstack:
        # ... existing configuration ...
        environment:
          - CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml
        volumes:
          - ./temporal-metrics.yaml:/etc/otelcol-contrib/custom.config.yaml:ro
          - ./temporal.key:/etc/otelcol-contrib/temporal.key:ro
          # ... other volumes ...
    ```

    <h5 id="all-in-one">
      Option 2: Docker run (all-in-one image)
    </h5>

    If using the all-in-one image with `docker run`:

    ```bash theme={null}
    docker run --name clickstack \
      -p 8080:8080 -p 4317:4317 -p 4318:4318 \
      -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
      -v "$(pwd)/temporal-metrics.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
      -v "$(pwd)/temporal.key:/etc/otelcol-contrib/temporal.key:ro" \
      clickhouse/clickstack-all-in-one:latest
    ```
  </Step>

  <Step>
    <h4 id="verifying-metrics">
      Verify metrics in HyperDX
    </h4>

    Once configured, log into HyperDX and verify metrics are flowing:

    1. Navigate to the Metrics explorer
    2. Search for metrics starting with `temporal` (e.g., `temporal_cloud_v1_workflow_success_count`, `temporal_cloud_v1_poll_timeout_count`)
    3. You should see metric data points appearing at your configured collection interval

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-1d264819/Fv_6myZEVt-yNIkA/images/clickstack/temporal/temporal-metrics.png?fit=max&auto=format&n=Fv_6myZEVt-yNIkA&q=85&s=e3fc466c9d4ef8b9fa7d981890f8b588" alt="Temporal Metrics" size="md" width="1133" height="628" data-path="images/clickstack/temporal/temporal-metrics.png" />
  </Step>
</Steps>

<h2 id="dashboards">
  Dashboards and visualization
</h2>

To help you get started monitoring Temporal Cloud with ClickStack, we provide some example visualizations for Temporal Metrics.

<Steps>
  <Step>
    <h4 id="download">
      <TrackedLink href={'/examples/temporal-metrics-dashboard.json'} download="temporal-metrics-dashboard.json" eventName="docs.temporal_metrics_monitoring.dashboard_download">Download</TrackedLink> the dashboard configuration
    </h4>
  </Step>

  <Step>
    <h4 id="import-dashboard">
      Import the pre-built dashboard
    </h4>

    1. Open HyperDX and navigate to the Dashboards section
    2. Click **Import Dashboard** in the upper right corner under the ellipses

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-1d264819/KBMpfHRObzlPpAN0/images/clickstack/import-dashboard.png?fit=max&auto=format&n=KBMpfHRObzlPpAN0&q=85&s=6e43e9f6e79839ce7706f4ab0f592964" alt="Import dashboard button" width="3024" height="556" data-path="images/clickstack/import-dashboard.png" />

    3. Upload the `temporal-metrics-dashboard.json` file and click **Finish Import**

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-1d264819/Fv_6myZEVt-yNIkA/images/clickstack/temporal/import-temporal-metrics-dashboard.png?fit=max&auto=format&n=Fv_6myZEVt-yNIkA&q=85&s=b171d345600a6c57dc54e408c0aaaa0c" alt="Finish import dialog" width="3600" height="2028" data-path="images/clickstack/temporal/import-temporal-metrics-dashboard.png" />
  </Step>

  <Step>
    <h4 id="created-dashboard">
      View the dashboard
    </h4>

    The dashboard will be created with all visualizations pre-configured:

    <Image img="https://mintcdn.com/private-7c7dfe99-mintlify-1d264819/Fv_6myZEVt-yNIkA/images/clickstack/temporal/temporal-metrics-dashboard.png?fit=max&auto=format&n=Fv_6myZEVt-yNIkA&q=85&s=692acd7857f02a5dfb9bd71d5a596906" alt="Temporal Metrics dashboard" width="3840" height="1933" data-path="images/clickstack/temporal/temporal-metrics-dashboard.png" />
  </Step>
</Steps>

<h2 id="troubleshooting">
  Troubleshooting
</h2>

<h3 id="troubleshooting-not-loading">
  Custom config not loading
</h3>

Verify the environment variable `CUSTOM_OTELCOL_CONFIG_FILE` is set correctly:

```bash theme={null}
docker exec <container-name> printenv CUSTOM_OTELCOL_CONFIG_FILE
```

Check that the custom config file is mounted at `/etc/otelcol-contrib/custom.config.yaml`:

```bash theme={null}
docker exec <container-name> ls -lh /etc/otelcol-contrib/custom.config.yaml
# usually, docker exec clickstack ls -lh /etc/otelcol-contrib/custom.config.yaml
```

View the custom config content to verify it's readable:

```bash theme={null}
docker exec <container-name> cat /etc/otelcol-contrib/custom.config.yaml
# usually, docker exec clickstack cat /etc/otelcol-contrib/custom.config.yaml
```

Confirm the `temporal.key` is mounted into the container:

```bash theme={null}
docker exec <container-name> cat /etc/otelcol-contrib/temporal.key
# usually, docker exec clickstack cat /etc/otelcol-contrib/temporal.key
# This should output your temporal.key
```

<h3 id="no-metrics">
  No metrics appearing in HyperDX
</h3>

Verify Temporal Cloud is accessible from the collector:

```bash theme={null}
# From the ClickStack container
docker exec <container-name> curl -H "Authorization: Bearer <API_KEY>" https://metrics.temporal.io/v1/metrics
```

You should see a series of Prometheus metrics printed e.g.

```text theme={null}
temporal_cloud_v1_workflow_success_count{operation="CompletionStats",region="aws-us-east-2",temporal_account="l2c4n",temporal_namespace="clickpipes-aws-prd-apps-us-east-2.l2c4n",temporal_task_queue="clickpipes-svc-dc118d12-b397-4975-a33e-c2888ac12ac4-peer-flow-task-queue",temporal_workflow_type="QRepPartitionWorkflow"} 0.067 1765894320
```

Verify the effective config includes your Prometheus receiver:

```bash theme={null}
docker exec <container> cat /etc/otel/supervisor-data/effective.yaml | grep -A 10 "Prometheus:"
## usually, docker exec clickstack cat /etc/otel/supervisor-data/effective.yaml | grep -A 10 "prometheus:"
```

Check for errors in the collector agent logs:

```bash theme={null}
docker exec <container> cat /etc/otel/supervisor-data/agent.log | grep -i Prometheus
# Look for connection errors or authentication failures
# docker exec clickstack cat /etc/otel/supervisor-data/agent.log | grep -i Prometheus
```

Check in the collector logs:

```bash theme={null}
docker exec <container> cat /var/log/otel-collector.log | grep -i error
# Look for config parsing errors - early supervisor.opamp-client can be ignored 
# docker exec clickstack cat /var/log/otel-collector.log | grep -i error
```

<h3 id="auth-errors">
  Authentication errors
</h3>

If you see authentication errors in the logs check your API key.

<h3 id="network-issues">
  Network connectivity issues
</h3>

If ClickStack can't reach Temporal Cloud ensure your Docker Compose file or `docker run` commands allow [external networking](https://docs.docker.com/engine/network/#drivers).

<h2 id="next-steps">
  Next steps
</h2>

* Set up [alerts](/clickstack/features/alerts) for critical metrics (workflow failure rates, task backlog growth, schedule-to-start latency)
* Create additional dashboards for specific use cases (namespace-level monitoring, workflow type performance)
* Monitor multiple Temporal Cloud accounts by duplicating the receiver configuration with different endpoints and service names

<h2 id="going-to-production">
  Going to production
</h2>

This guide extends ClickStack's built-in OpenTelemetry Collector for quick setup. For production deployments, we recommend running your own OTel Collector and sending data to ClickStack's OTLP endpoint. See [Sending OpenTelemetry data](/clickstack/ingesting-data/opentelemetry) for production configuration.