Backstage has become the go-to open-source framework for internal developer portals — and for good reason. With its plugin architecture, Software Catalog, and support for service ownership, it centralizes metadata that teams usually spread across half a dozen tools.

But there’s still friction. Searching for the right service, understanding deployment status, or locating a runbook can take way too many clicks. What if instead of navigating, developers could just ask?

In this article, we’ll walk through how to bring LLM-powered search and summarization into your Backstage portal — enabling natural language queries like:

• “Who owns the checkout-service and when was it last deployed?”
• “Where is the runbook for restarting the Kafka cluster?”
• “What changed before the incident last night?”

We’ll use OpenAI for language understanding, ChromaDB for semantic search, and a lightweight backend service to orchestrate retrieval and prompting.

Architecture Overview

We’ll break it down into three layers:

1. Data Layer: Backstage’s Software Catalog, docs, deployment logs, and service metadata
2. Index + Search Layer: Embedding and storing relevant content in a vector DB
3. LLM Orchestration Layer: Accepts a natural language query, performs vector search, builds a prompt, and returns an answer

Step 1: Extracting Data from Backstage

You’ll need to tap into:

• Backstage’s /entities API: returns metadata for services, systems, APIs, etc.
• TechDocs (if enabled): markdown documentation for each service
• Additional plugins or custom APIs for:
  • CI/CD logs
  • On-call schedules
  • Deployment events

Example: Pulling the Software Catalog

curl http://localhost:7007/api/catalog/entities | jq

Save this as backstage_metadata.json.

Each service has keys like:

{
  "kind": "Component",
  "metadata": {
    "name": "checkout-service",
    "annotations": {
      "github.com/project-slug": "my-org/checkout-service",
      "pagerduty.com/service-id": "abc123"
    }
  },
  "spec": {
    "owner": "team-payments",
    "type": "service",
    "lifecycle": "production"
  }
}

Step 2: Embedding and Indexing with ChromaDB

Use OpenAI’s text-embedding-3-small to vectorize relevant fields like descriptions, owners, and annotations.

import openai
import chromadb
import json

openai.api_key = os.getenv("OPENAI_API_KEY")

def embed_text(text):
    response = openai.Embedding.create(
        model="text-embedding-3-small",
        input=text
    )
    return response['data'][0]['embedding']

client = chromadb.Client()
collection = client.create_collection("backstage-services")

with open("backstage_metadata.json") as f:
    services = json.load(f)

for svc in services:
    name = svc['metadata']['name']
    owner = svc['spec'].get('owner', 'unknown')
    desc = svc['metadata'].get('description', '')
    doc = f"{name} is owned by {owner}. {desc}"
    embedding = embed_text(doc)
    collection.add(documents=[doc], embeddings=[embedding], ids=[name])

Step 3: Query Interface + Prompting

Now we build a simple FastAPI backend to accept a user query and return a natural language answer.

query_handler.py

from fastapi import FastAPI, Request
from pydantic import BaseModel
import openai
import chromadb

openai.api_key = os.getenv("OPENAI_API_KEY")

client = chromadb.Client()
collection = client.get_collection("backstage-services")

app = FastAPI()

class Query(BaseModel):
    question: str

@app.post("/ask")
def ask(query: Query):
    results = collection.query(query_texts=[query.question], n_results=5)
    context = "\n".join(results['documents'][0])

    prompt = f"""
You are a helpful internal dev assistant. Based on the following service 
metadata, answer the user's question:

User: {query.question}
Metadata:
{context}
"""

    response = openai.ChatCompletion.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )

    return {"answer": response['choices'][0]['message']['content']}

Step 4: Embedding It in Backstage

Backstage plugins can host custom frontends. You could add a simple “Ask” page with a chat input box that POSTs to your /ask endpoint and displays the result.

Use @backstage/core-plugin-api and React to embed a chat UI or button like:

<Button onClick={submitQuery}>Ask the Portal</Button>

Or add a global search provider that integrates semantic suggestions into your command bar.

Bonus: Conversation + Memory

You can extend this with:

• LangChain or LlamaIndex to add memory across user sessions
• RAG chaining (Retrieve-Augment-Generate) across multiple data sources
• Slackbot integration using the same backend
• GitHub issue lookup with “what PR closed this ticket?”

Challenges and Cautions

• Index freshness: periodically re-ingest and re-embed your service metadata
• Prompt tuning: LLMs can hallucinate if metadata is vague
• Security: Don’t expose secrets in deployment logs or configs
• Latency: Cache popular responses if needed

Conclusion

With just a bit of effort, you can turn Backstage from a structured metadata dashboard into a conversational, AI-powered developer assistant. Instead of digging through tabs and clicking through links, your developers get immediate, contextual answers — and get back to shipping faster.

Next up: let’s build a full-stack prototype using LangChain for natural language developer queries, and make this flow available outside the portal, in Slack, CLI tools, or even VS Code.

Want more hands-on LLM-powered dev tooling? Explore our articles on Slaptijack