Azure Content Understanding

https://learn.microsoft.com/en-us/azure/ai-services/content-understanding
Admin
Azure Content Understanding is a solution that analyzes and comprehends various media content...
Tokens:194,667
Snippets:1,301
Trust Score:10
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Install azure-ai-contentunderstanding

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/quickstart/use-rest-api

Install the Content Understanding client library for Python with pip. Use the --pre flag to install the pre-release version.

```console
pip install --pre azure-ai-contentunderstanding
```

--------------------------------

### Install Content Understanding .NET package

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/how-to/copy-analyzers

Installs the Content Understanding client library for .NET.

```console
dotnet add package Azure.AI.ContentUnderstanding

```

--------------------------------

### Install Content Understanding client library

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/quickstart/use-rest-api

Installs the Content Understanding client library for .NET as a prerelease package.

```console
dotnet add package Azure.AI.ContentUnderstanding --prerelease

```

--------------------------------

### Install NuGet packages for Azure.AI.ContentUnderstanding

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/how-to/classification-content-understanding-studio

Install the required NuGet packages for the .NET Content Understanding client.

```bash
dotnet add package Azure.AI.ContentUnderstanding
dotnet add package Azure.Identity

```

--------------------------------

### Analyze audio with prebuilt-audioSearch analyzer (Java)

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/quickstart/use-rest-api

Java example using the prebuilt-audioSearch analyzer to extract an audio transcript, generate a summary, and perform speaker labeling. Requires CONTENTUNDERSTANDING_ENDPOINT and CONTENTUNDERSTANDING_KEY environment variables. The output shows the markdown transcript, a summary, and the first two transcript phrases with speaker and start time.

```java
import java.util.Arrays;
import com.azure.core.credential.AzureKeyCredential;
import com.azure.core.util.polling.SyncPoller;
import com.azure.ai.contentunderstanding.ContentUnderstandingClient;
import com.azure.ai.contentunderstanding.ContentUnderstandingClientBuilder;
import com.azure.ai.contentunderstanding.models.*;

public class test_audio {

    public static void main(String[] args) {
        String endpoint = System.getenv("CONTENTUNDERSTANDING_ENDPOINT");
        String key = System.getenv("CONTENTUNDERSTANDING_KEY");

        ContentUnderstandingClient client =
            new ContentUnderstandingClientBuilder()
                .endpoint(endpoint)
                .credential(new AzureKeyCredential(key))
                .buildClient();

        AnalysisInput input = new AnalysisInput();
        input.setUrl(
            "https://raw.githubusercontent.com/"
            + "Azure-Samples/"
            + "azure-ai-content-understanding-assets/"
            + "main/audio/callCenterRecording.mp3"
        );

        SyncPoller<ContentAnalyzerAnalyzeOperationStatus, AnalysisResult> operation =
            client.beginAnalyze(
                "prebuilt-audioSearch",
                Arrays.asList(input)
            );

        AnalysisResult result = operation.getFinalResult();

        // Cast to AudioVisualContent for audio-specific properties
        AudioVisualContent audioContent =
            (AudioVisualContent) result.getContents().get(0);
        System.out.println(audioContent.getMarkdown());

        String summary = audioContent.getFields() != null
            && audioContent.getFields().containsKey("Summary")
            ? audioContent.getFields().get("Summary")
                .getValue().toString()
            : "";
        System.out.println("Summary: " + summary);

        if (audioContent.getTranscriptPhrases() != null
            && !audioContent.getTranscriptPhrases().isEmpty()) {
            System.out.println("Transcript (first two phrases):");
            int count = 0;
            for (TranscriptPhrase phrase
                : audioContent.getTranscriptPhrases()) {
                if (count >= - break;
                System.out.println(
                    "  [" + phrase.getSpeaker() + "] "
                    + phrase.getStartTime().toMillis()
                    + " ms: " + phrase.getText()
                );
                count++;
            }
        }
    }
}

```

```text
# Audio: 00:00.000 => 00:32.183

Transcript

WEBVTT

00:00.080 --> 00:00.640
<v Speaker 1>Good day.

00:00.880 --> 00:02.240
<v Speaker 1>Welcome to Contoso.

00:02.560 --> 00:03.760
<v Speaker 1>My name is John Doe.

00:03.920 --> 00:05.120
<v Speaker 1>How can I help you today?

00:05.440 --> 00:06.320
<v Speaker 2>Yes, good day.

00:06.640 --> 00:08.160
<v Speaker 2>My name is Maria Smith.

00:08.560 --> 00:11.360
<v Speaker 2>I would like to inquire about my current point balance.  

00:11.680 --> 00:12.560
<v Speaker 1>No problem.

00:12.880 --> 00:13.920
<v Speaker 1>I am happy to help.

00:14.240 --> 00:16.720
<v Speaker 1>I need your date of birth to confirm your identity.      

00:17.120 --> 00:19.600
<v Speaker 2>It is April 19th, 1988.

00:20.000 --> 00:20.480
<v Speaker 1>Great.

00:20.800 --> 00:24.160
<v Speaker 1>Your current point balance is 599 points.

00:24.560 --> 00:26.160
<v Speaker 1>Do you need any more information?

00:26.480 --> 00:27.200
<v Speaker 2>No, thank you.

00:27.600 --> 00:28.320
<v Speaker 2>That was all.

00:28.720 --> 00:29.360
<v Speaker 2>Goodbye.

00:29.680 --> 00:31.920
<v Speaker 1>You're welcome, goodbye a Cantoso.

Summary: The conversation is a customer service interaction where Maria Smith contacts Contoso to inquire about her current point balance. The agent, John Doe, verifies her identity by asking for her date of birth and then provides her with the information that she has 599 points. The customer confirms that she does not need any further information and ends the call politely.
Transcript (first two phrases):
  [Speaker 1] 80 ms: Good day.
  [Speaker 1] 880 ms: Welcome to Contoso.

```

--------------------------------

### Install MarkItDown with Content Understanding support

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/integrations/markitdown

Use this command to install MarkItDown with the optional dependency for Azure Content Understanding support.

```bash
pip install 'markitdown[az-content-understanding]'

```

--------------------------------

### Install TypeScript and Content Understanding client library

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/quickstart/use-rest-api

Installs TypeScript and the Content Understanding client library.

```console
npm install typescript ts-node @azure/ai-content-understanding@next

```

--------------------------------

### Install required packages for Python sample

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/how-to/classification-content-understanding-studio

Install the azure-ai-contentunderstanding prerelease build (1.2.0b2 or later) which includes the to_llm_input helper, along with azure-identity and python-dotenv.

```bash
pip install --pre azure-ai-contentunderstanding
pip install azure-identity python-dotenv
```

--------------------------------

### Install Content Understanding SDK

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/how-to/copy-analyzers

Installs the Content Understanding client library for JavaScript. Required to use the SDK in your project.

```Console
npm install @azure/ai-content-understanding

```

--------------------------------

### Install azure-ai-contentunderstanding

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/how-to/copy-analyzers

Install the Content Understanding client library for Python using pip. This is required before you can use the SDK to copy custom analyzers.

```bash
pip install azure-ai-contentunderstanding

```

--------------------------------

### Install agent-framework-azure-contentunderstanding with pip

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/integrations/agent-framework

Installs the Content Understanding integration package for the Microsoft Agent Framework. The --pre flag installs the current prerelease build of the package.

```bash
pip install agent-framework-azure-contentunderstanding --pre

```

--------------------------------

### Install Azure.Identity package

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/how-to/copy-analyzers

Optionally installs the Azure Identity library for Microsoft Entra authentication.

```console
dotnet add package Azure.Identity

```

--------------------------------

### Install .NET SDK: dotnet add package Azure.AI.ContentUnderstanding

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/whats-new

Install the .NET SDK package for Content Understanding. Use this command to add the SDK to your .NET project.

```bash
dotnet add package Azure.AI.ContentUnderstanding
```

--------------------------------

### Create Maven project

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/quickstart/use-rest-api

Creates a new Maven project for the Content Understanding quickstart. Run this command in a terminal to generate the project structure and navigate into the project directory.

```console
mvn archetype:generate -DgroupId=com.example \
    -DartifactId=content-understanding-quickstart \
    -DarchetypeArtifactId=maven-archetype-quickstart \
    -DinteractiveMode=false
cd content-understanding-quickstart

```

--------------------------------

### Camera shots JSON example

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/video/elements

Example of cameraShotTimesMs, which identifies points where camera shots change. The array stores the starting times of all shots except the first (which starts at 0 ms). This output is included when "returnDetails": true is set in the analyzer definition.

```json
{
  "cameraShotTimesMs": [
    2002,
    22356,
    26960,
    53353,
    71071,
    76210,
    78111,
    113487,
    148882,
    152953
  ]
}
```

--------------------------------

### Install required packages for Content Understanding samples

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/how-to/classification-content-understanding-studio

Install the required packages for the JavaScript sample: the Content Understanding client, Azure identity and core-auth for credentials, dotenv for environment variables, and TypeScript types for development.

```Bash
npm install @azure/ai-content-understanding @azure/identity @azure/core-auth dotenv
npm install --save-dev typescript @types/node

```

--------------------------------

### Example response with supportedModels for prebuilt-invoice

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/concepts/models-deployments

Example response from the GET analyzers API showing the supportedModels object with completion and embedding models, and the models section using aliases for a prebuilt analyzer.

```json
{
  "analyzerId": "prebuilt-invoice",
  "supportedModels": {
    "completion": [
      "gpt-5.2"
    ],
    "embedding": [
      "text-embedding-3-large"
    ]
  },
  "models": {
    "completion": "prebuilt-analyzer-completion",
    "embedding": "prebuilt-analyzer-embedding"
  }
}

```

--------------------------------

### Create a new .NET console application

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/quickstart/use-rest-api

Creates a new .NET console application named ContentUnderstandingQuickstart and navigates into its directory.

```console
dotnet new console -n ContentUnderstandingQuickstart
cd ContentUnderstandingQuickstart

```

--------------------------------

### Chart.js bar chart JSON (chartFormat=chartJs)

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/document/markdown

Chart.js JSON configuration for a bar chart. Used when `config.chartFormat` is set to `chartJs`. The example shows a bar chart with monthly labels, dataset values, and a y-axis starting at zero.

```json
{
  "type": "bar",
  "data": {
    "labels": ["Jan", "Feb", "Mar", "Apr", "May", "Jun"],
    "datasets": [{
      "label": "Values",
      "data": [100, 150, 200, 250, 300, 275],
      "backgroundColor": "rgba(54, 162, 235, 0.2)",
      "borderColor": "rgba(54, 162, 235, 1)",
      "borderWidth": 1
    }]
  },
  "options": {
    "scales": {
      "y": {
        "beginAtZero": true
      }
    }
  }
}
```

--------------------------------

### GET response with analysis result

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/quickstart/use-rest-api

Example GET response showing a successful analysis result with status "Succeeded" and extracted fields in the result object. If the operation isn't complete, status is "Running" or "NotStarted"; poll every 1-2 seconds until "Succeeded". To include optional model-call and latency information, add the x-ms-diagnostics: true header to every polling request.

```jsonc
{
  "id": "ce05fb5a-579e-4f0b-afb5-3532bcddeaee",
  "status": "Succeeded",
  "result": {
    "analyzerId": "prebuilt-invoice",
    "apiVersion": "2025-11-01",
    "createdAt": "2025-11-13T20:04:55Z",
    "warnings": [],
    "contents": [
      {
        "path": "input1",
        "markdown": "CONTOSO LTD.\n\n\n# INVOICE\n\nContoso Headquarters\n123 456th St\nNew York, NY, 10001\n\nINVOICE: INV-100\n\nINVOICE DATE: 11/15/2019\n\nDUE DATE: 12/15/2019\n\nCUSTOMER NAME: MICROSOFT CORPORATION...",
        "fields": {
          "AmountDue": {
            "type": "object",
            "valueObject": {
              "Amount": {
```

--------------------------------

### Install JavaScript/TypeScript SDK: npm install @azure/ai-content-understanding

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/whats-new

Install the JavaScript/TypeScript SDK package for Content Understanding. Use this command to add the SDK to your Node.js project.

```bash
npm install @azure/ai-content-understanding
```

--------------------------------

### Install Python SDK: pip install azure-ai-contentunderstanding

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/whats-new

Install the Python SDK package for Content Understanding. Use this command to add the SDK to your Python environment.

```bash
pip install azure-ai-contentunderstanding
```

--------------------------------

### Install langchain-azure-ai package

Source: https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/integrations/langchain

Installs the langchain-azure-ai package, which contains the AzureAIContentUnderstandingLoader.

```bash
pip install -U langchain-azure-ai

```
Tokens:
Raw