Soul Machines

https://docs.soulmachines.com
Admin
Soul Machines provides an AI platform and developer tools, including a Web SDK and Skills API, for...
Tokens:105,286
Snippets:965
Trust Score:9.8
Update:1 month ago
Show doc for...
Tokens:
Raw
### Usage of setMediaDeviceActive with microphone and camera

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene.setmediadeviceactive

Usage example that starts the microphone and stops the camera. Note the missing parentheses in the then callback and the missing closing parenthesis in the catch callback as shown in the source.

```javascript
scene.setMediaDeviceActive({ microphone: true, camera: false })  
  .then(console.log('microphone activated, camera deactivated'));  
  .catch((error) => console.log('error occurred: ', error);  

```

--------------------------------

### Connection State Data Example

Source: https://docs.soulmachines.com/web-sdk/guides/connection-state

Example of the connectionStateData structure received in the callback, including percentageLoaded and currentStep values.

```text
    name: Disconnected  
    currentStep: 0  
    totalSteps: 5  
    percentageLoaded: 0  
  
    name: SearchingForDigitalPerson  
    currentStep: 1  
    totalSteps: 5  
    percentageLoaded: 25  
  
    name: DownloadingAssets  
    currentStep: 2  
    totalSteps: 5  
    percentageLoaded: 50  
  
    name: ConnectingToDigitalPerson  
    currentStep: 3  
    totalSteps: 5  
    percentageLoaded: 75  
  
    name: Connected  
    currentStep: 4  
    totalSteps: 5  
    percentageLoaded: 100  
```

--------------------------------

### Start Python app

Source: https://docs.soulmachines.com/skills-api/getting-started/nlp-adapter-skill

Run the Python application.

```bash
python3 app.py run
```

--------------------------------

### Start video with scene.startVideo()

Source: https://docs.soulmachines.com/web-sdk/guides/video-autoplay

Calls the provided startVideo() function on a scene to begin video playback. It automatically mutes the video if audio is not allowed, and returns a videoState object or throws an error if playback cannot start.

```javascript
scene  
  .startVideo()  
  .then((videoState) => console.log('started video with state:', videoState))  
  .catch((error) => console.log('could not start video:', error));  

```

--------------------------------

### startSpeaking request/response example

Source: https://docs.soulmachines.com/docs/websocket-message-formats

Example request/response pair for the startSpeaking message. The request includes a transaction ID (1) and a body with personaId, text, and context. The response echoes the transaction and includes a status value where >= 0 indicates success.

```json
{  
    "category": "scene",  
    "kind":"request",  
    "name": "startSpeaking",  
    "transaction":  1, // null | <string|number> - if null the request is one-way and no response will be sent   
    "body":   
    {  
        "personaId": <string|number>,  
        "text" : "Blah blah",  
        "context": "..."    // a context string e.g. node or intentName   
        // that will be included in the transcript   
    }  
}  
  
{  
    "category":     "scene",  
    "kind":         "response",  
    "name":         "startSpeaking",  
    "transaction":  1,  
    "status": 0, // ‘C’ sdk status value where >= 0 is success, < 0 is failure  
    "body":   
    {  
    }  
}  
  

```

--------------------------------

### Proxy video connect button and scene setup

Source: https://docs.soulmachines.com/web-sdk/guides/video-autoplay

Shows a proxy video setup: a hidden in-memory video is used to connect to the scene, and its stream is attached to a visible video element. The connect button handler must not be async to preserve the user gesture token required for autoplay.

```html
<button id="connectButton" autoplay>Connect</button>  
  
<video id="video" autoplay></video>  

```

```javascript
const scene = new Scene();  
const connectButton = document.getElementById('connectButton');  
const renderedVideo = document.getElementById('video');  
const proxyVideo = document.createElement('video');  
  
  
const playVideo = () => {  
  // Make sure we are testing with audio unmuted  
  renderedVideo.muted = false;  
  
  // Attach video stream to our rendered video  
  renderedVideo.srcObject = videoStream;  
  
  videoRef.currentrenderedVideo  
      .play()  
      .then(() => {  
        // Can play with audio  
        renderedVideo.muted = false;  
      })  
      .catch(() => {  
        // Can't play with audio, so mute it  
        renderedVideo.muted = true;  
      })  
}  
  
// It is important NOT to make this function asynchronous with the async keyword. You'll lose the "user gesture token" required to allow your video to play later.  
const connectToScene = () => {  
  // Tell browser we're loading a video  
  renderedVideo.load();  
  
  // Connect to video source from `scene` and fire `playVideo` once we're connected  
  scene.connect().then(() => playVideo());  
}  
  
connectButton.addEventListener('click', connectToScene)  

```

--------------------------------

### Install SkillSDK for NodeJS

Source: https://docs.soulmachines.com/skills-api/getting-started/nlp-adapter-skill

Install the Soul Machines SkillSDK package using npm.

```bash
npm i @soulmachines/smskillsdk
```

--------------------------------

### Install SkillSDK for Python

Source: https://docs.soulmachines.com/skills-api/getting-started/nlp-adapter-skill

Install the Soul Machines SkillSDK package using pip.

```bash
pip install smskillsdk
```

--------------------------------

### startRecognize

Source: https://docs.soulmachines.com/docs/websocket-message-formats

Starts the speech recognizer. Relies on the persona having a configured speech recognizer.

```APIDOC
## startRecognize

### Description
Starts the speech recognizer. Relies on the persona having a configured speech recognizer.

### Method
WebSocket Message

### Endpoint
WebSocket message with category "scene", kind "request", name "startRecognize"

### Parameters
#### Request Body
- **category** (string) - Required - Must be "scene"
- **kind** (string) - Required - Must be "request"
- **name** (string) - Required - Must be "startRecognize"
- **transaction** (null|string|number) - Required - If null the request is one-way and no response will be sent
- **body.audioSource** (integer) - Optional - 2 (processed) or 4 (squelched), defaults to processed

### Request Example
```json
{
    "category": "scene",
    "kind": "request",
    "name": "startRecognize",
    "transaction": null,
    "body": {
        "audioSource": 2
    }
}
```
```

--------------------------------

### Install smskillsdk with npm

Source: https://docs.soulmachines.com/skills-api/sdks/node-sdk

Install the @soulmachines/smskillsdk package. Requires NodeJS 14 and above.

```bash
npm install @soulmachines/smskillsdk  
```

--------------------------------

### Usage of startVideo

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene.startvideo

Example showing how to call startVideo and handle potential audio muting or user interaction requirements.

```javascript
scene.startVideo()  
     .then(({ video, audio }) => {  
        if (!audio) {  
         //video is muted, ask user to unmute video  
        }  
     })  
     .catch((error) => {  
        if (error.name === 'userInteractionRequired') {  
         //ask user to interact with the UI  
         //unmute video and play again  
         video.muted = false;  
         video.play();  
        }  
     });  
```

--------------------------------

### startRecognize(audioSource)

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene

Starts the speech to text recognizer.

```APIDOC
## startRecognize(audioSource)

### Description
Starts the speech to text recognizer.

### Method
startRecognize

### Parameters
- **audioSource** (MediaStreamTrack) - Required - The audio source to recognize.

### Returns
Promise<void> - Resolves when recognition starts.
```

--------------------------------

### Private Config String Example

Source: https://docs.soulmachines.com/skills-api/api-reference/skill-definition

Defines a private configuration entry with a string value.

```json
{
  "name": "my_private_string",
  "value": "string"
}
```

--------------------------------

### Handle Welcome intent in /execute endpoint (ExpressJS and Python)

Source: https://docs.soulmachines.com/skills-api/getting-started/nlp-adapter-skill

Both examples implement the /execute endpoint for a Soul Machines skill, handling the optional "Welcome" message by sending a platform-specific initialization message (e.g., 'START') instead of the literal text. They extract credentials from the skill config, forward user input to a fake NLP service, and return an ExecuteResponse with the spoken text and endConversation set to true. The TypeScript version uses Express and the Python version uses FastAPI.

```typescript
app.post('/execute', async (req: Request, res: Response) => {  
  // Get the Soul Machines request object  
  const smRequest = req.body as ExecuteRequest;  
  
  // Extract relevant data (eg credentials) from skill config  
  const { firstCredentials, secondCredentials } = smRequest.config;  
  
  // Get the user's input text  
  const userInput = smRequest.text;  
  
  // @TODO: Replace this with a connection to your own NLP Platform  
  const fakeNLPService = new FakeNLPService(  
    firstCredentials,  
    secondCredentials  
  );  
  
  // differentiate between "Welcome" message and actual user input  
  let fakeNLPServiceResponse;  
  if (userInput === 'Welcome') {  
    // TODO: send a conversation initialization message  
    // in the correct format for your NLP Platform.  
    // eg. if your "fakeNLPService" expected a "START" message  
    // then it might look something like this.  
    fakeNLPServiceResponse = await fakeNLPService.send('START');  
  } else {  
    fakeNLPServiceResponse = await fakeNLPService.send(userInput);  
  }  
  
  // extract relevant data from the third-party response  
  const spokenResponse = fakeNLPServiceResponse.text;  
  
  // Construct SM-formatted response body  
  const smResponse: ExecuteResponse = {  
    output: {  
      text: spokenResponse,  
    },  
    endConversation: true,  
  };  
  
  res.send(smResponse);  
});  
```

```python
@router.post("/execute", status_code=200, response_model=ExecuteResponse, response_model_exclude_unset=True)  
async def execute(request: ExecuteRequest) -> ExecuteResponse:  
    # Get the skill config from request object  
    skill_config = request.config  
  
    # Extract relevant data (eg credentials) from skill config  
    credentials = itemgetter("first_credentials", "second_credentials")(skill_config)  
  
    # Get the user's input text  
    user_input = request.text  
  
    # @TODO: Replace this with a connection to your own NLP Platform  
    fake_nlp_service = FakeNLPService(*credentials)  
  
    # differentiate between "Welcome" message and actual user input  
    fake_nlp_response: str  
    if user_input === "Welcome":  
        # TODO: send a conversation initialization message  
        # in the correct format for your NLP Platform.  
        # eg. if your "fake_nlp_service" expected a "START" message  
        # then it might look something like this.  
        fake_nlp_response = fake_nlp_service.send("START")  
    else:  
        fake_nlp_response = fake_nlp_service.send(user_iInput)  
  
    # Extract relevant data from the third-party response  
    spoken_response = fake_nlp_response.text  
  
    # Construct SM-formatted response body  
    output = Output(  
        text=spoken_response,  
    )  
  
    response = ExecuteResponse(  
        output=output,  
        endConversation=True,  
    )  
  
    return response  
```

--------------------------------

### Start ExpressJS app

Source: https://docs.soulmachines.com/skills-api/getting-started/nlp-adapter-skill

Run the ExpressJS application on a specified port.

```bash
PORT=3000 npm start
```

--------------------------------

### Dropdown Config Example

Source: https://docs.soulmachines.com/skills-api/api-reference/skill-definition

Defines a dropdown selection input field with predefined options.

```json
{
  "label": "My Dropdown",
  "name": "my_dropdown",
  "type": "SELECT",
  "options": [
    {
      "text": "None",
      "value": ""
    },
    {
      "text": "My Option 1",
      "value": "option_1"
    }
  ],
  "required": false,
  "toolTipText": "**Supports Markdown** - Basic text _styling_ and [links](http://google.com)"
}
```

--------------------------------

### Example event message

Source: https://docs.soulmachines.com/docs/websocket-message-formats

Example of an event message sent from the Persona to all connected servers and the connected client web browser. Shows the standard event fields: category, kind, name, and body.

```json
{  
    "category": "scene",  
    "kind": "event",  
    "name": "state",  
    "body": {  
        "persona": {  
            "1": { // is a personaId  
                "speechState": "idle",  
                "currentSpeech": "Blah blah"  
            }  
        }  
    }  
}  

```

--------------------------------

### Private Config Array Example

Source: https://docs.soulmachines.com/skills-api/api-reference/skill-definition

Defines a private configuration entry with an array value.

```json
{
  "name": "my_private_array",
  "value": ["string"]
}
```

--------------------------------

### Password Config Example

Source: https://docs.soulmachines.com/skills-api/api-reference/skill-definition

Defines a password input field with length validation constraints.

```json
{
  "label": "My Password",
  "name": "my_password"
  "type": "PASSWORD",
  "hint": "Hint for this config",
  "placeholder": "Enter Password Here",
  "required": true,
  "validationWarning": "Must be between 1 - 10 characters.",
  "validationMin": 1,
  "validationMax": 10,
  "toolTipText": "**Supports Markdown** - Basic text _styling_ and [links](http://google.com)"
}
```

--------------------------------

### Text Config Example

Source: https://docs.soulmachines.com/skills-api/api-reference/skill-definition

Defines a text input field for skill configuration with validation rules and markdown-supported tooltips.

```json
{
  "label": "My Text",
  "name": "my_text",
  "type": "TEXT",
  "placeholder": "",
  "defaultValue": "default text",
  "required": false,
  "validationWarning": "Can only contain letters",
  "validationRegex": "^[a-zA-Z]+$",
  "toolTipText": "**Supports Markdown** - Basic text _styling_ and [links](http://google.com)"
}
```

--------------------------------

### Usage of isMicrophoneActive

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene.ismicrophoneactive

Example showing how to call the isMicrophoneActive method on a scene instance.

```javascript
const isMicrophoneActive = scene.isMicrophoneActive();
```

--------------------------------

### Usage of isCameraActive

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene.iscameraactive

Example showing how to call the isCameraActive method on a scene instance.

```javascript
const isCameraActive = scene.isCameraActive();  
```

--------------------------------

### Config Overwrite Example

Source: https://docs.soulmachines.com/skills-api/api-reference/skill-definition

Demonstrates how configMeta takes precedence over privateConfig when name properties conflict.

```json
  configMeta: {
    "name": "my_string",
    "label": "My String",
    "type": "TEXT"
  },
  privateConfig: {
    "name": "my_string",
    "value": "string"
  }
```

--------------------------------

### configMeta parameter examples for DDNA Studio skills

Source: https://docs.soulmachines.com/skills-api/getting-started/registering-your-skill

These JSON objects define the configMeta parameters for DDNA Studio skills. Each object specifies a name, type, label, required flag, description, and optional fields like placeholder, defaultValue, and toolTipText. The examples cover TEXT, TEXT_AREA, PASSWORD, NUMBER, SELECT, and BOOLEAN types, with usage notes such as the placeholder being ignored for SELECT and BOOLEAN types.

```json
{  
    "name": "no_match",  
    "type": "TEXT",  
    "label": "No Match Response",  
    "required": false,  
    "description": "Phrase the digital person will say if no match was found for the user's query",  
    "placeholder": "Enter your personalized no match phrase here",  
    "defaultValue": "Sorry, I don't have an answer for that",  
    "toolTipText": "**Supports Markdown** - Basic text _styling_ and [links](http://google.com)"  
}  

```

```json
{  
    "name": "prompt_hook",  
    "type": "TEXT_AREA",  
    "label": "Additional Prompting",  
    "required": false,  
    "description": "Any conversation goals, rules, or restrictions?",  
    "placeholder": "e.g., I'm the Soul Machines Influencer and I love everything about my job! I keep my responses conversational, brief, and accurate",  
    "toolTipText": "**Supports Markdown** - Basic text _styling_ and [links](http://google.com)"  
}  

```

```json
{  
    "name": "user_password",  
    "type": "PASSWORD",  
    "label": "PASSWORD",  
    "required": false,  
    "description": "Password required to access premium services",  
    "placeholder": "Please enter your password here",  
    "toolTipText": "**Supports Markdown** - Basic text _styling_ and [links](http://google.com)"  
}  

```

```json
{  
    "name": "config_sleep_time",  
    "type": "NUMBER",  
    "label": "Wait time before going to sleep",  
    "required": false,  
    "description": "Enter the wait time in minutes before the digital human goes to sleep due to inactivity",  
    "placeholder": "3",  
    "defaultValue": "3",  
}  

```

```json
{  
    "name": "spoken_language",  
    "type": "SELECT",  
    "label": "Language that the digital human will speak",  
    "options": [  
        {  
            "text": "English",  
            "value": "en"  
        },  
        {  
            "text": "Japanese",  
            "value": "ja"  
        },  
        {  
            "text": "Korean",  
            "value": "jo"  
        },  
        {  
            "text": "Arabic",  
            "value": "ar"  
        }  
    ],  
    "required": false,  
    "defaultValue": "en"  
}  

```

```json
{  
    "name": "enable_transcription",  
    "type": "BOOLEAN",  
    "label": "Enable transcription services",  
    "required": false,  
    "description": "If enabled, conversations will be transcribed.",  
    "defaultValue": "false"  
}  

```

--------------------------------

### Scene.setMediaDeviceActive()

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene.setmediadeviceactive

Starts or stops streaming video/audio to the server based on the values of `microphone` and `camera`.

```APIDOC
## setMediaDeviceActive(options)

### Description
On success, starts or stops streaming video/audio to the server based on the values of `microphone` and `camera`.

### Method
setMediaDeviceActive

### Signature
```
setMediaDeviceActive(options: {  
        microphone?: boolean;  
        camera?: boolean;  
    }): Promise<void>;  
```

### Parameters
- **options** (object) - Required - An object with optional `microphone` and `camera` boolean properties.

### Returns
`Promise<void>` - A promise which is fulfilled when the media active state has been successfully changed. If the session is not defined it will return `undefined`. If the active state could not be changed, the promise is rejected with an Error object having the format:
```
{  
  message: string;  
  name: errorCode;  
}  
```

Where `errorCode` is one of:
- `noUserMedia` - the microphone/camera is either not available, not usable or the user declined permission to use them
- `failedUpgrade` - the media upgrade failed
- `notSupported` - user’s browser does not support the getUserMedia API
- `noConnection` - connection has not been established - ensure scene.connect() has been called previously

### Usage Example
```
scene.setMediaDeviceActive({ microphone: true, camera: false })  
  .then(console.log('microphone activated, camera deactivated'));  
  .catch((error) => console.log('error occurred: ', error);  
```
```

--------------------------------

### Text Area Config Example

Source: https://docs.soulmachines.com/skills-api/api-reference/skill-definition

Defines a multi-line text area input field for skill configuration.

```json
{
  "label": "My Text Area",
  "name": "my_text_area",
  "type": "TEXT_AREA",
  "placeholder": "",
  "defaultValue": "default text",
  "required": false,
  "validationWarning": "Can only contain letters",
  "validationRegex": "^[a-zA-Z]+$",
  "toolTipText": "**Supports Markdown** - Basic text _styling_ and [links](http://google.com)"
}
```

--------------------------------

### SessionStart Message Payload

Source: https://docs.soulmachines.com/skills-api/api-reference/async-api

Example payload for a sessionStart message sent to the skill at the beginning of a session.

```json
{
  "projectId": "string",
  "sessionId": "string",
  "config": {},
  "deploymentEnvironment": "preview"
}
```

--------------------------------

### Watson Assistant Options Card JSON

Source: https://docs.soulmachines.com/web-sdk/guides/building-content-cards

Example of a user-defined options card structure for Watson Assistant.

```json
{
		"user_defined": {
			"id": "sampleOptionsCard",
			"data": {
				"options": [
					{
						"label": "One",
						"value": "value one"
					},
					{
						"label": "Two",
						"value": "value two"
					},
					{
						"label": "Three",
						"value": "value three"
					}
				]
			},
			"type": "options"
		},
		"response_type": "user_defined"
	}
```

--------------------------------

### Handle NO_MATCH in Python

Source: https://docs.soulmachines.com/skills-api/getting-started/nlp-adapter-skill

This Python example shows how to return a NO_MATCH intent when the third-party NLP service provides no response.

```python
@router.post("/execute", status_code=200, response_model=ExecuteResponse, response_model_exclude_unset=True)  
async def execute(request: ExecuteRequest) -> ExecuteResponse:  
    # Get the skill config from request object  
    skill_config = request.config  
  
    # Extract relevant data (eg credentials) from skill config  
    credentials = itemgetter("first_credentials", "second_credentials")(skill_config)  
  
    # Get the user's input text  
    user_input = request.text  
  
    # @TODO: Replace this with a connection to your own NLP Platform  
    fake_nlp_service = FakeNLPService(*credentials)  
    fake_nlp_response = fake_nlp_service.send(user_iInput);  
  
    # Extract relevant data from the third-party response  
    spoken_response = fake_nlp_response.text  
  
    # Construct base SM-formatted response body  
    response = ExecuteResponse(  
        endConversation=True,  
    )  
    # Example of a possible NO_MATCH when no response is returned by third-party  
    if spoken_response: # only set text output if there is a response  
      response.output = Output(  
        text=spoken_response,  
      )  
    else: # set intent to NO_MATCH  
      response.intent = Intent(  
        name="NO_MATCH",  
        confidence=1, # or as defined by the third-party  
      )  
  
    return response  
```

--------------------------------

### Implement execute endpoint handler

Source: https://docs.soulmachines.com/skills-api/sdks/python-sdk

Example of a handler function for generating an ExecuteResponse and a corresponding FastAPI route method.

```python
# execute endpoint handler containing response generation logic  
def execute_handler(request: ExecuteRequest) -> ExecuteResponse:  
  
    # response generation logic here  
  
    variables = Variables(public={ "card": { ... }})  
  
    output = Output(  
        text="",  
        variables=variables  
    )  
  
    response = ExecuteResponse(  
        output=output,  
        memory=[],  
        endConversation=True,  
    )  
  
    return response  
  
# route method (using FastAPI syntax)  
@app.post("/execute", status_code=status.HTTP_200_OK, response_model=ExecuteResponse, response_model_exclude_unset=True)  
def execute(request: ExecuteRequest):  
    return execute_handler(request)
```

--------------------------------

### getTranscript

Source: https://docs.soulmachines.com/docs/websocket-message-formats

Gets transcript records for the session that describe interactions between the user and the persona. Each record includes a millisecond timestamp since the start of the session and either recognized user speech or persona state information. Records are requested with a starting cursor position (an integer id) and all records are returned since that cursor position to the time in which the request is made.

```APIDOC
## getTranscript

### Description
Gets transcript records for the session that describe interactions between the user and the persona. Each record includes a millisecond timestamp since the start of the session and either recognized user speech or persona state information. Records are requested with a starting cursor position (an integer id) and all records are returned since that cursor position to the time in which the request is made.

### Method
WebSocket Message

### Endpoint
WebSocket message with category "scene", kind "request", name "getTranscript"

### Parameters
#### Request Body
- **category** (string) - Required - Must be "scene"
- **kind** (string) - Required - Must be "request"
- **name** (string) - Required - Must be "getTranscript"
- **transaction** (string|number) - Required - Transaction identifier
- **body.cursor** (integer) - Optional - Starting cursor position, defaults to 0

### Request Example
```json
{
    "category": "scene",
    "kind": "request",
    "name": "getTranscript",
    "transaction": "<string|number>",
    "body": {
        "cursor": 0
    }
}
```

### Response
#### Success Response
- **body.nextCursor** (integer) - Next cursor position
- **body.startCursor** (integer) - Starting cursor position
- **body.records** (array) - Array of transcript records
  - **timestamp** (integer) - Milliseconds since start of session
  - **recognizeResults** (object) - Same content as recognize results, only successful final results are included
  - **state** (object) - Same content as persona state event, only changed portions of 'persona' object are included

#### Response Example
```json
{
    "body": {
        "nextCursor": 5,
        "startCursor": 0,
        "records": [
            {
                "timestamp": 0,
                "recognizeResults": {}
            },
            {
                "timestamp": 2000,
                "state": {}
            },
            {
                "timestamp": 5000,
                "recognizeResults": {}
            }
        ]
    }
}
```
```

--------------------------------

### Scene Instantiation with Configuration

Source: https://docs.soulmachines.com/web-sdk/guides/scene-configurations

Creates a new Scene instance with an API key, video element, and requested/required media devices. The requestedMediaDevices and requiredMediaDevices are shown with microphone and camera set to true.

```javascript
  scene = new Scene({  
    apiKey: apiKey,  
    videoElement: videoEl,  
    requestedMediaDevices: { microphone: true, camera: true },  
    requiredMediaDevices: { microphone: true, camera: true },  
  });  

```

--------------------------------

### Create skill-definition.json

Source: https://docs.soulmachines.com/skills-api/getting-started/nlp-adapter-skill

Initial configuration file for the Base Skill.

```json
{
  "name": "My NLP Adapter",
  "summary": "Connects a third-party platform as a conversation provider",
  "description": "",
  "status": "ACTIVE",
  "serviceProvider": "SKILL_API",
  "config": {
    "skillType": "BASE_CORPUS"
  }
}
```

--------------------------------

### get onSpeechMarkerEvent(): SmEvent

Source: https://docs.soulmachines.com/web-sdk/guides/persona-configuration

Gets the event that fires when a speech marker is received.

```APIDOC
## Accessor: onSpeechMarkerEvent

### Description
Gets the event that fires when a speech marker is received.

### Method
Accessor

### Endpoint
get onSpeechMarkerEvent(): SmEvent

### Returns
`SmEvent`
```

--------------------------------

### get onConversationResultEvent(): SmEvent

Source: https://docs.soulmachines.com/web-sdk/guides/persona-configuration

Gets the event that fires when a conversation result is received.

```APIDOC
## Accessor: onConversationResultEvent

### Description
Gets the event that fires when a conversation result is received.

### Method
Accessor

### Endpoint
get onConversationResultEvent(): SmEvent

### Returns
`SmEvent`
```

--------------------------------

### Execute Response Sample

Source: https://docs.soulmachines.com/skills/api

Example 200 OK response from the /execute endpoint, containing the output text, variables, intent, memory, and conversation control flags.

```json
{
  "output": {
    "text": "Here is a cat",
    "variables": {
      "public": {
        "exitVariable": true,
        "card": {
          "type": "image",
          "data": {
            "url": "https://placekitten.com/200/200",
            "alt": "An adorable kitten"
}
}
}
}
},
  "intent": {
    "name": "MY_INTENT",
    "confidence": 0
},
  "memory": [
    {
      "name": "memory1",
      "scope": "PRIVATE",
      "value": "value1",
      "session_id": "456ABC456"
},
    {
      "name": "memory2",
      "scope": "PUBLIC",
      "value": {
        "name": "string"
}
},
    {
      "name": "memory3",
      "scope": "PUBLIC",
      "value": 1,
      "session_id": "123ABC123"
}
],
  "endConversation": true,
  "endRouting": false

}
```

--------------------------------

### Configuring logging in Scene constructor

Source: https://docs.soulmachines.com/web-sdk/guides/logging

Example of initializing the Scene with custom logging settings, enabling debug session logs and disabling content awareness logging.

```javascript
const scene = new smwebsdk.Scene(  
  video,  
  false,  
  UserMedia.MicrophoneAndCamera,  
  UserMedia.MicrophoneAndCamera,  
  300,  
  {  
    session: {  
      enabled: true,  
      minLogLevel: 'debug',  
    },  
    contentAwareness: {  
      enabled: false  
    }  
  },  
);  
```

--------------------------------

### Usage example for supportsSessionPersistence()

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene.supportssessionpersistence

Example of calling supportsSessionPersistence() on a Scene instance to check if session persistence is supported. The result is stored in a boolean variable.

```typescript
const isSessionPersistenceSupported = scene.supportsSessionPersistence();
```

--------------------------------

### Dialogflow optionalArgs example

Source: https://docs.soulmachines.com/docs/websocket-message-formats

Example 'optionalArgs'/query parameters for Dialogflow. Includes kind, contexts (with name, lifespanCount, parameters), payload, timeZone, geoLocation, and sentimentAnalysisRequestConfig.

```json
{  
    "kind": "text"/"event"/"init",  
    "contexts":   
    [ {   
        "name": "contextId", // This may be provided in its full form e.g. "projects/<project_id>/agent/sessions/<session_id>/contexts/contextId" or just context ID without the prefix                                                  
        "lifespanCount": 100,                                                                
        "parameters":   
        {  
            "name": "Bob"   
        }  
    } ],                                                                   
    "payload": { "foo": 1},                                                                  
    "timeZone": "America/New_York",                                                                 
    "geoLocation":   
    {  
        "latitude": 51.42,  
        "longitude": 9.999  
    },  
    "sentimentAnalysisRequestConfig":  
    {  
        "analyzeQueryTextSentiment": true                                                                }  
}  

```

--------------------------------

### get onStateEvent(): SmEvent

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene.onstateevent

Gets the SmEvent instance for scene state events. This event can be used to register listeners that fire when scene state changes occur.

```APIDOC
## get onStateEvent(): SmEvent

### Description
Returns the SmEvent object that emits state events for the Scene. Use this event's `.on()` method to subscribe to scene state changes.

### Signature
```
get onStateEvent(): SmEvent;
```

### Returns
- **SmEvent** - The event object for subscribing to scene state events.
```

--------------------------------

### Scene Instantiation with Element

Source: https://docs.soulmachines.com/web-sdk/guides/scene-configurations

Creates a new Scene instance using an HTML element reference. This is a minimal example that does not specify additional configuration options.

```javascript
const scene = new Scene(element);  

```

--------------------------------

### ContentAwareness.setupEventListeners()

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.contentawareness.setupeventlisteners

Sets up event listeners for the ContentAwareness instance. This method initializes the necessary event handling for content awareness features.

```APIDOC
## ContentAwareness.setupEventListeners()

### Description
Sets up event listeners for the ContentAwareness instance. This method initializes the necessary event handling for content awareness features.

### Signature
```
setupEventListeners(): void;
```

### Returns
void
```

--------------------------------

### Scene.startVideo()

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene.startvideo

Documentation for the Scene.startVideo() method in the @soulmachines/smwebsdk SDK. This method plays the video element and returns a promise with video and audio status, handling browser autoplay restrictions.

```APIDOC
## Scene.startVideo()

### Description
Plays the video element and returns results. Different browsers have different restrictions on autoplay. Using this method can handle all the cases browsers can have on initial video playback.

### Method
startVideo

### Signature
```typescript
startVideo(videoElement?: HTMLVideoElement): Promise<{ video: boolean; audio: boolean; } | Error>;
```

### Parameters
- **videoElement** (HTMLVideoElement) - Optional - Optional parameter specifying the video element hosting the Digital Person. If not specified, the video element passed to the Scene constructor will be used.

### Returns
Promise<{ video: boolean; audio: boolean; } | Error>

Returns a promise which is fulfilled when the video playback is successful, with indication of video and audio status. If the video element is not defined or video play fails, the promise is rejected with an Error object having the format:

```typescript
{
  message: string;
  name: errorCode;
}
```

Where `errorCode` is one of:
- `noVideoElement` - no HTMLVideoElement found from `videoElement` or `Scene` constructor
- `userInteractionRequired` - cannot start media playback due to browser restriction; user interaction is required before playing again

### Usage Example
```typescript
scene.startVideo()
     .then(({ video, audio }) => {
        if (!audio) {
         //video is muted, ask user to unmute video
        }
     })
     .catch((error) => {
        if (error.name === 'userInteractionRequired') {
         //ask user to interact with the UI
         //unmute video and play again
         video.muted = false;
         video.play();
        }
     });
```
```

--------------------------------

### Example color memory object

Source: https://docs.soulmachines.com/skills-api/guides/memory

Shows a concrete JavaScript object literal for a Memory with the name 'favorite-color', the value 'blue', a session_id of 'xxx-xxx-xxx', and scope 'PUBLIC'. This example illustrates the expected shape for creating a memory.

```javascript
const colorMemory = {  
  name: 'favorite-color',  
  value: 'blue',  
  session_id: 'xxx-xxx-xxx',  
  scope: 'PUBLIC',  
};  
```

--------------------------------

### constructor(videoElement?, audioOnly?, requestedUserMedia?, requiredUserMedia?, contentAwarenessDebounceTime?, loggingConfig?, tracerOptions?, chromaKeyOptions?)

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene._constructor__1

Constructs a Scene instance with optional parameters. This constructor is obsolete; use `new Scene(options: SceneOptions)` instead.

```APIDOC
## Scene Constructor

### Description
Constructs a Scene with parameters. This API is obsolete; use `new Scene(options: SceneOptions)` instead.

### Signature

constructor(videoElement?: HTMLVideoElement, audioOnly?: boolean, requestedUserMedia?: UserMedia, requiredUserMedia?: UserMedia, contentAwarenessDebounceTime?: number, loggingConfig?: Partial<LoggingConfig>, tracerOptions?: TracerOptions, chromaKeyOptions?: ChromaKeyOptions);


### Parameters
- **videoElement** (HTMLVideoElement) - Optional - A video element that will display the connected scene.
- **audioOnly** (boolean) - Optional - This streaming should be audio streaming only (no video streaming).
- **requestedUserMedia** (UserMedia) - Optional - The user media devices (microphone/camera) that should be requested, one of: UserMedia.None, UserMedia.Microphone, UserMedia.MicrophoneAndCamera (default).
- **requiredUserMedia** (UserMedia) - Optional - Required user media devices, one of: UserMedia.None, UserMedia.Microphone, UserMedia.MicrophoneAndCamera. If less user media devices are requested than required, the requirements take precedence. If this user media requirement is not met, Connect() will fail.
- **contentAwarenessDebounceTime** (number) - Optional - The timeout period used for debouncing messaging within the content awareness class.
- **loggingConfig** (Partial<LoggingConfig>) - Optional - Options to configure different log levels for different classes.
- **tracerOptions** (TracerOptions) - Optional - Tracer options.
- **chromaKeyOptions** (ChromaKeyOptions) - Optional - Chroma key options.

### Usage Note
This constructor is obsolete. Use `new Scene(options: SceneOptions)` instead.
```

--------------------------------

### getState()

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene

Gets the current scene state.

```APIDOC
## getState()

### Description
Gets the current scene state.

### Method
getState

### Parameters
None.

### Returns
SceneState - The current scene state.
```

--------------------------------

### connect(serverUri, userText, accessToken, retryOptions)

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene

Connects to a scene at the given server URI.

```APIDOC
## connect(serverUri, userText, accessToken, retryOptions)

### Description
Connects to a scene at the given server URI.

### Method
connect

### Parameters
- **serverUri** (string) - Required - The server URI.
- **userText** (string) - Required - User text.
- **accessToken** (string) - Required - Access token.
- **retryOptions** (object) - Required - Retry options.

### Returns
Promise<void> - Resolves when connected.
```

--------------------------------

### SceneResponseError.message

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.sceneresponseerror.message

Gets the error message associated with the SceneResponseError.

```APIDOC
## SceneResponseError.message property

### Description
Gets the error message associated with the SceneResponseError.

### Signature
```typescript
get message(): string;
```

### Returns
- **string** - The error message.
```

--------------------------------

### Scene.connect() method

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene.connect_1

Documentation for the Scene.connect() method, including its signature, parameters, and return value.

```APIDOC
## Scene.connect() method

### Description
Connects to a scene at the given server URI. Returns a promise that holds success/failure callbacks. If the promise is rejected, an Error is given as the argument, which converts to a string message.

### Signature
```
connect(serverUri?: string, userText?: string, accessToken?: string, retryOptions?: RetryOptions): Promise<string | undefined>;
```

### Parameters
- **serverUri** (string) - Optional - The server websocket URI to connect to.
- **userText** (string) - Optional - A custom text string that is sent to the orchestration server.
- **accessToken** (string) - Optional - A JWT access token issued to permit access to this server.
- **retryOptions** (RetryOptions) - Optional - Options for customizing connection error retry.

### Returns
- **Promise<string | undefined>** - A promise that holds success/failure callbacks. If the promise is rejected, an Error is given as the argument, which converts to a string message. The error result has two fields: 'message' (string) and 'name' (one of the following error name/reason codes):
  - **notSupported** - The browser does not support getUserMedia.
  - **noUserMedia** - The microphone/camera is either not available, not usable, or the user declined permission to use them.
  - **serverConnectionFailed** - The connection to the server failed.
  - **noScene** - No persona was available.
  - **mediaStreamFailed** - The audio/video stream failed.
  - **sessionTimeout** - The session timed out before it was fully available.
```

--------------------------------

### Scene.currentPersonaId

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene.currentpersonaid

Gets the current persona ID of the scene.

```APIDOC
## Scene.currentPersonaId property

### Description
Gets the current persona ID of the scene.

### Signature
```
currentPersonaId: PersonaId;
```

### Property Value
- **Type**: PersonaId
- **Description**: The identifier of the current persona.
```

--------------------------------

### Implement NLP Adapter endpoint in Python

Source: https://docs.soulmachines.com/skills-api/getting-started/nlp-adapter-skill

This Python example demonstrates using FastAPI to handle an /execute POST request, extracting credentials, calling a third-party service, and returning a structured ExecuteResponse.

```python
@router.post("/execute", status_code=200, response_model=ExecuteResponse, response_model_exclude_unset=True)  
async def execute(request: ExecuteRequest) -> ExecuteResponse:  
    # Get the skill config from request object  
    skill_config = request.config  
  
    # Extract relevant data (eg credentials) from skill config  
    credentials = itemgetter("first_credentials", "second_credentials")(skill_config)  
  
    # Get the user's input text  
    user_input = request.text  
  
    # @TODO: Replace this with a connection to your own NLP Platform  
    fake_nlp_service = FakeNLPService(*credentials)  
    fake_nlp_response = fake_nlp_service.send(user_input)  
  
    # Extract relevant data from the third-party response  
    spoken_response = fake_nlp_response.text  
  
    # Construct SM-formatted response body  
    output = Output(  
        text=spoken_response,  
    )  
  
    response = ExecuteResponse(  
        output=output,  
        endConversation=True,  
    )  
  
    return response  
```

--------------------------------

### Persona.conversationGetVariables()

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.persona

Gets the current conversation variable values.

```APIDOC
## Persona.conversationGetVariables()

### Description
Get variables will return the current conversation variable values in the promise completion. Completion/promise receives an object with member 'variables'.

### Method
conversationGetVariables

### Parameters
None

### Returns
Promise - A promise that resolves with an object containing a 'variables' member.
```

--------------------------------

### Handle connection rejection when device required

Source: https://docs.soulmachines.com/web-sdk/guides/accessing-camera-and-microphone

Wrap the Scene creation with .catch to handle the error when user denies access. Log or display message.

```javascript
const scene = new Scene({  
    videoElement: videoEl,  
    requestedMediaDevices: { microphone: true },  
    requiredMediaDevices: { microphone: true },  
  })  
  .then(() => console.log('connected!'))  
  .catch(() => {  
      console.log('user declined device access');  
      // show a message to the user explaining  
      // that they must approve media device access  
  });  

```

--------------------------------

### configure(configuration)

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene

Sends configuration to the scene.

```APIDOC
## configure(configuration)

### Description
Sends configuration to the scene.

### Method
configure

### Parameters
- **configuration** (object) - Required - The configuration to send.

### Returns
Promise<void> - Resolves when configuration is sent.
```

--------------------------------

### Scene.onRecognizeResultsEvent

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene.onrecognizeresultsevent

Gets the event that is triggered when recognition results are available.

```APIDOC
## Scene.onRecognizeResultsEvent

### Description
Gets the event that is triggered when recognition results are available.

### Method
Property getter

### Signature
```
get onRecognizeResultsEvent(): SmEvent;
```

### Returns
- **SmEvent** - The event object for recognition results.
```

--------------------------------

### Scene.onCameraActive

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene.oncameraactive

Gets the SmEvent that is triggered when the camera becomes active.

```APIDOC
## Scene.onCameraActive property

### Description
Gets the SmEvent that is triggered when the camera becomes active.

### Signature
```
get onCameraActive(): SmEvent;
```

### Returns
- **SmEvent** - The event object that can be used to subscribe to camera activation events.
```

--------------------------------

### onConversationStateUpdated

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.conversationstate.onconversationstateupdated

Gets the event that is triggered when the conversation state changes.

```APIDOC
## onConversationStateUpdated

### Description
Gets the event that is triggered when the conversation state changes.

### Method
Property getter

### Signature
```
get onConversationStateUpdated(): SmEvent;
```

### Returns
- **SmEvent** - The event object that can be used to subscribe to conversation state updates.
```

--------------------------------

### Implement Session endpoint in Python

Source: https://docs.soulmachines.com/skills-api/guides/session-state-management

Example implementation of the session endpoint using Python to handle session initialization and memory persistence.

```python
@router.post("/session", status_code=200, response_model=SessionResponse, response_model_exclude_unset=True)
async def session(request: SessionRequest) -> SessionResponse:
  # Extract relevant data such as config, memory and sessionId as required
  skill_config, skill_memory, session_id = attrgetter("config", "memory", "sessionId")(request)

  # TODO: Preload your Skill's required resources before each session starts, as required
  fake_nlp_service = FakeNLPService()
  fake_nlp_service.init_session_resources(session_id, skill_config, skill_memory)

  # TODO: Persist data in Memory as required
  data_to_persist = fake_nlp_service.persist_data()
  set_memory_value(memories=skill_memory, data_to_persist)

  # Construct SM-formatted response body
  response = SessionResponse(
    memory=skill_memory,
    # ... add your other required kwargs here
  )

  return response
```

--------------------------------

### onConnectionStateUpdated

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.connectionstate.onconnectionstateupdated

Gets the event that is triggered when the connection state is updated.

```APIDOC
## onConnectionStateUpdated

### Description
Gets the event that is triggered when the connection state is updated.

### Method
Property getter

### Signature
```
get onConnectionStateUpdated(): SmEvent;
```

### Returns
- **SmEvent** - The event object that can be used to subscribe to connection state updates.
```

--------------------------------

### connect(options)

Source: https://docs.soulmachines.com/web-sdk/api/smwebsdk.scene

Connects to a scene using an options object.

```APIDOC
## connect(options)

### Description
Connects to a scene using an options object.

### Method
connect

### Parameters
- **options** (object) - Required - Connection options.

### Returns
Promise<void> - Resolves when connected.
```

--------------------------------

### configure

Source: https://docs.soulmachines.com/docs/websocket-message-formats

Configures the scene. This request is atomic - if one item cannot be applied, none are applied and the request fails.

```APIDOC
## configure

### Description
Configures the scene. This request is atomic - if one item cannot be applied, none are applied and the request fails.

### Direction
Server to Scene

### Message Format
```json
{
    "category": "scene",
    "kind": "request",
    "name": "configure",
    "transaction": null | <string|number>,
    "body": {
        "directionalLight": {
            "x": 0.0,
            "y": 0.0,
            "z": 0.0
        },
        "audio": {
            "squelchLevel": -30.0
        },
        "persona": {
            "<personaId>": {
                "conversation": {
                    "speakResults": true
                }
            }
        },
        "Statistics": {
            "intervalSeconds": 1.0
        }
    }
}
```

### Parameters
#### Body
- **directionalLight** (object) - Optional - Contains `x`, `y`, `z` floats for light direction.
- **audio.squelchLevel** (float) - Optional - Audio squelch level.
- **persona.<personaId>.conversation.speakResults** (bool) - Optional - Defaults to true.
- **Statistics.intervalSeconds** (float) - Optional - Statistics reporting interval.

### Notes
- All properties in the body are optional.
- If `transaction` is null, the request is one-way and no response will be sent.
```
Tokens:
Raw