Building a Backstage Service for Custom OTel Metrics Export

About Backstage

Backstage is an open-source platform for building developer portals, originally developed by Spotify. It streamlines infrastructure tooling, services, and documentation into a single, consistent UI, making it easier for developers to navigate complex environments. Backstage is now a Cloud Native Computing Foundation (CNCF) incubating project, which underscores its growing adoption and community support.

Backstage, developed by Spotify, is an open-source platform for building developer portals. While it offers robust features out of the box, you might need to extend its functionality to suit your specific needs — especially when it comes to observability and monitoring. In this article, we’ll walk through the process of creating a Backstage service that sends custom OpenTelemetry (OTel) metrics data.

By the end of this tutorial, you’ll have a working Backstage service that pushes mock OTel metrics data to an OpenTelemetry Collector via gRPC.

Prerequisites

Before you begin, ensure you have the following:

1. An existing Backstage backend setup.
2. Basic knowledge of Backstage plugins and services.
3. A running OpenTelemetry Collector, either locally or remotely.
4. Installed @opentelemetry packages.

Make sure you have the following @opentelemetry packages installed in your Backstage project:

yarn add @opentelemetry/api @opentelemetry/sdk-metrics @opentelemetry/exporter-metrics-otlp-grpc @grpc/grpc-js

Step 1: Create the OpenTelemetry Metrics Service

We’ll start by creating a new service that sends mock metrics data to the OpenTelemetry Collector.

Create the Service File

Navigate to your Backstage project directory and create a new file for the service:

touch packages/backend/src/plugins/otel-metrics-service.ts

Now, add the following code to otel-metrics-service.ts:

import { diag, DiagConsoleLogger, DiagLogLevel } from '@opentelemetry/api';
import { MeterProvider } from '@opentelemetry/sdk-metrics';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-grpc';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import * as grpc from '@grpc/grpc-js';

export class OtelMetricsService {
  static start() {
    diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.ALL);
    const metricExporter = new OTLPMetricExporter({
      url: 'grpc://localhost:4317',
      credentials: grpc.credentials.createInsecure(),
    });
    const meterProvider = new MeterProvider();
    meterProvider.addMetricReader(
      new PeriodicExportingMetricReader({
        exporter: metricExporter,
        exportIntervalMillis: 10000,
      }),
    );
    const meter = meterProvider.getMeter('backstage-metrics');
    const usersGauge = meter.createObservableGauge('users_engaged', {
      description: 'Number of users engaged with Backstage in the last 28 days',
    });
    usersGauge.addCallback((observableResult) => {
      observableResult.observe(350);
    });
    const weeklySearchesGauge = meter.createObservableGauge('weekly_searches', {
      description: 'Number of searches in the last week',
    });
    weeklySearchesGauge.addCallback((observableResult) => {
      observableResult.observe(1565);
    });
    const weeklyUsersGauge = meter.createObservableGauge('weekly_users', {
      description: 'Number of weekly active users',
    });
    weeklyUsersGauge.addCallback((observableResult) => {
      observableResult.observe(505);
    });
    const templateRunsGauge = meter.createObservableGauge('template_runs', {
      description: 'Number of template runs in the last 28 days',
    });
    templateRunsGauge.addCallback((observableResult) => {
      observableResult.observe(7);
    });
    const feedbackCounter = meter.createCounter('feedback_items', {
      description: 'Number of feedback items collected',
    });
    feedbackCounter.add(2);
    const requestCounter = meter.createCounter('test_requests', {
      description: 'Test request counter',
    });
    setInterval(() => {
      requestCounter.add(1);
      console.log('Test metric recorded');
    }, 10000);
  }
}

Explanation

In this code, we’re creating mock metrics for users, searches, and feedback. These metrics will be sent to the OpenTelemetry Collector every 10 seconds using an insecure gRPC connection, assuming the collector is running locally.

Configuring the Metric Exporter

const metricExporter = new OTLPMetricExporter({
  url: 'grpc://localhost:4317',
  credentials: grpc.credentials.createInsecure(),
});

This exporter is configured to send metrics to the OpenTelemetry Collector running locally on port 4317 using gRPC.

Creating the Meter Provider

const meterProvider = new MeterProvider();

The `MeterProvider` is responsible for managing metric instruments and collecting metric data.

Adding a Metric Reader

meterProvider.addMetricReader(
  new PeriodicExportingMetricReader({
    exporter: metricExporter,
    exportIntervalMillis: 10000,
  }),
);

A `PeriodicExportingMetricReader` is added to the meter provider, which periodically exports collected metrics every 10,000 milliseconds (10 seconds) using the configured exporter.

Pushing mock data to OTel

Pushing Engaged data

const usersGauge = meter.createObservableGauge('users_engaged', {
  description: 'Number of users engaged with Backstage in the last 28 days',
});
usersGauge.addCallback((observableResult) => {
  observableResult.observe(350);
});

This gauge reports the number of users engaged over the last 28 days, set to a constant value of 350.

Pushing Weekly Searches data

const weeklySearchesGauge = meter.createObservableGauge('weekly_searches', {
  description: 'Number of searches in the last week',
});
weeklySearchesGauge.addCallback((observableResult) => {
  observableResult.observe(1565);
});

Reports a constant value of 1,565 searches in the last week.

Pushing Weekly Users data

const weeklyUsersGauge = meter.createObservableGauge('weekly_users', {
  description: 'Number of weekly active users',
});
weeklyUsersGauge.addCallback((observableResult) => {
  observableResult.observe(505);
});

Reports 505 weekly active users.

Pushing Template Runs data

const templateRunsGauge = meter.createObservableGauge('template_runs', {
  description: 'Number of template runs in the last 28 days',
});
templateRunsGauge.addCallback((observableResult) => {
  observableResult.observe(7);
});

Reports 7 template runs over the last 28 days

Step 2: Integrate the Service into Backstage

Next, we’ll integrate the service into the Backstage backend. We’ll initialize the service in the index.ts file where the backend starts.

Modify index.ts

In your packages/backend/src/index.ts, import the service and call its start() method when the backend initializes:

import { createBackend } from '@backstage/backend-defaults';
import { OtelMetricsService } from './plugins/otel-metrics-service';

const backend = createBackend();
OtelMetricsService.start();
backend.add(import('@backstage/plugin-app-backend/alpha'));
backend.add(import('@backstage/plugin-proxy-backend/alpha'));
backend.add(import('@backstage/plugin-scaffolder-backend/alpha'));
backend.add(import('@backstage/plugin-techdocs-backend/alpha'));
// Add other plugins as needed...
backend.start();

By adding OtelMetricsService.start(), you ensure that the OpenTelemetry metrics service starts when the Backstage backend does. It will periodically export mock metrics to your collector.

Step 3: Set Up the OpenTelemetry Collector

You’ll need an OpenTelemetry Collector to receive and process the metrics. If you don’t have one running locally, you can start a basic setup using Docker:

docker run --rm -p 4317:4317 -p 9464:9464 \
  -v "$(pwd)/otel-collector-config.yaml":/otel-collector-config.yaml \
  otel/opentelemetry-collector:latest \
  --config /otel-collector-config.yaml

This command assumes you have a configuration file named otel-collector-config.yaml in your current directory. If you don't, here's a simple configuration for testing purposes:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"  

processors:
  batch:  

exporters:
  logging:  
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

Step 4: Run the Backstage Backend

With the service integrated, you can now run the Backstage backend:

yarn start

You should see output indicating that metrics are being recorded and exported:

Test metric recorded

Console screenshot : Backstage

Check your OpenTelemetry Collector or Prometheus/Grafana dashboards to ensure the metrics are being received and processed.

Step 5: Verify Metrics in Prometheus or Grafana

To visualize or monitor these metrics, you can integrate Prometheus or Grafana with your OpenTelemetry setup. Configure the OpenTelemetry Collector to export metrics to Prometheus, and then visualize the metrics in Grafana.

Integrating OpenTelemetry with Backstage enhances your application’s observability by providing detailed metrics that can be monitored and analyzed. In this article, we’ve walked through creating a custom service within Backstage to export mock metrics to an OpenTelemetry Collector using gRPC. This foundational setup allows you to extend and customize metrics collection according to your application’s needs.

For further reading and to deepen your understanding, consider exploring the following resources:

OpenTelemetry Documentation:

• OpenTelemetry Official Site
• OpenTelemetry JavaScript SDK
• OpenTelemetry Collector
• OpenTelemetry Metrics API

Backstage Documentation:

• Backstage Official Website
• Backstage GitHub Repository
• Backstage Plugins and Integrations
• Getting Started with Backstage