https://github.com/neocortex-link/neocortex-unity-sdk
Neocortex Unity SDK for Smart NPCs and Virtual Assistants
https://github.com/neocortex-link/neocortex-unity-sdk
ai ai-agent ai-agents ai-tools aiagent aiagents conversational-ai game-ai game-development multimodal-ai npc npcs smart-agent smart-agents smart-npc smart-npcs
Last synced: 3 months ago
JSON representation
Neocortex Unity SDK for Smart NPCs and Virtual Assistants
- Host: GitHub
- URL: https://github.com/neocortex-link/neocortex-unity-sdk
- Owner: neocortex-link
- License: mit
- Created: 2024-10-30T22:26:30.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2025-02-21T14:43:06.000Z (8 months ago)
- Last Synced: 2025-02-21T15:31:40.837Z (8 months ago)
- Topics: ai, ai-agent, ai-agents, ai-tools, aiagent, aiagents, conversational-ai, game-ai, game-development, multimodal-ai, npc, npcs, smart-agent, smart-agents, smart-npc, smart-npcs
- Language: C#
- Homepage: https://neocortex.link
- Size: 227 KB
- Stars: 7
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE.md
Awesome Lists containing this project
README
## Neocortex Unity SDK
The Neocortex Unity SDK is a Unity package that allows you to easily integrate Neocortex into your Unity project.
The SDK provides a set of APIs that allow you to interact with the projects created on the Neocortex web platform.
You can find more about the Unity SDK integration here in our documentatons: https://neocortex.link/docs/integrations/unity/quick-start
## Requirements
- Neocortex account [Sign Up Here](https://neocortex.link/register)
- Unity 2021.3 or above [Download Here](https://unity3d.com/get-unity/download)
- Git version control system [Download Here](https://git-scm.com/download)
## Installation
- Open your Unity project
- Go to `Window` > `Package Manager`
- Click on the `+` button and select `Add package from git URL`
- Paste the following URL: `https://github.com/neocortex-link/neocortex-unity-sdk.git`
- Click on the `Add` button
## Setup
### Save API Key
To start using the Neocortex SDK, you need to initialize it with your Neocortex API key. You can create a new API key from the Neocortex web platform by going to the [API Keys](https://neocortex.link/dashboard/api-keys) page.
- Create a new API key and copy it
- Open your Unity project
- Go to `Tools` > `Neocortex Settings`
- Paste the API key in the `API Key` field and click on the `Save` button
### Create a new Neocortex project
- Go to the [Neocortex web platform](https://neocortex.link/dashboard/projects) and create a new project
- Copy the project ID from the project details page
- Open your Unity project and go to your scene
- Create an empty GameObject and add the `Neocortex Smart Agent` component to it
- Paste the project ID in the `Project ID` field
## API Reference
After setting up the Neocortex SDK in your Unity project, you can start using the APIs to interact with the Neocortex project.
### Neocortex Smart Agent component
The `Neocortex Smart Agent` component is the main component that allows you to interact with the Neocortex project.
**public async void TextToText(string message)**
- Send a text message to the Neocortex project, and expect a text response.
- Parameters:
- `message`: The text message to send.
- Example:
```csharp
var smartAgent = GetComponent();
smartAgent.OnChatResponseReceived.AddListener((response) =>
{
Debug.Log($"Message: {response.message}");
Debug.Log($"Action: {response.action}");
});
smartAgent.TextToText("Hello, Neocortex!");
```
**public async void TextToAudio(string message)**
- Send a text message to the Neocortex project, and expect a audio response.
- Parameters:
- `message`: The text message to send.
- Example:
```csharp
var audioSource = GetComponent();
var smartAgent = GetComponent();
smartAgent.OnChatResponseReceived.AddListener((response) =>
{
Debug.Log($"Message: {response.message}");
Debug.Log($"Action: {response.action}");
});
smartAgent.OnAudioResponseReceived.AddListener((audioClip) =>
{
audioSource.clip = audioClip;
audioSource.Play();
});
smartAgent.TextToAudio("Hello, Neocortex!");
```
**public async void AudioToText(AudioClip audio)**
- Sends an audio clip to the Neocortex project. This method is used with `NeocortexAudioReceiver` component to send audio data.
- Parameters:
- `audioClip`: The audio clip to send.
- Example:
```csharp
var smartAgent = GetComponent();
smartAgent.OnTranscriptionReceived.AddListener((message) =>
{
Debug.Log($"You: {message}");
});
var audioReceiver = GetComponent();
audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
{
Debug.Log($"Audio Data Length: {audioClip.samples}");
smartAgent.AudioToText(audioClip);
});
// Start recording audio for 3 seconds
audioReceiver.StartMicrophone();
await Task.Delay(3000);
audioReceiver.StopMicrophone();
```
**public async void AudioToAudio(AudioClip audio)**
- Sends an audio clip to the Neocortex project and expects an audio response. This method is used with `NeocortexAudioReceiver` component to send audio data.
- Parameters:
- `audioClip`: The audio clip to send.
- Example:
```csharp
var audioSource = GetComponent();
var smartAgent = GetComponent();
smartAgent.OnAudioResponseReceived.AddListener((audioClip) =>
{
audioSource.clip = audioClip;
audioSource.Play();
});
smartAgent.OnTranscriptionReceived.AddListener((message) =>
{
Debug.Log($"You: {message}");
});
smartAgent.OnChatResponseReceived.AddListener((response) =>
{
Debug.Log($"Message: {response.message}");
Debug.Log($"Action: {response.action}");
});
var audioReceiver = GetComponent();
audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
{
Debug.Log($"Audio Data Length: {audioClip.samples}");
smartAgent.AudioToAudio(audioClip);
});
// Start recording audio for 3 seconds
audioReceiver.StartMicrophone();
await Task.Delay(3000);
audioReceiver.StopMicrophone();
```
**public UnityEvent OnChatResponseReceived**
- Event that is triggered when the Neocortex project responds to a text message.
- Parameters:
- `response`: The response from the Neocortex project.
- Example:
```csharp
var smartAgent = GetComponent();
smartAgent.OnChatResponseReceived += (response) =>
{
Debug.Log($"Message: {response.message}");
Debug.Log($"Action: {response.action}");
};
```
**public UnityEvent OnTranscriptionReceived**
- Event that is triggered when the Neocortex project transcribes an audio message to text.
- Parameters:
- `message`: The transcribed audio message.
- Example:
```csharp
var smartAgent = GetComponent();
smartAgent.OnTranscriptionReceived += (message) =>
{
Debug.Log($"You: {message}");
};
```
**public UnityEvent OnAudioResponseReceived**
- Event that is triggered when the Neocortex project responds with an audio message.
- Parameters:
- `audioClip`: The audio clip received from the Neocortex project.
- Example:
```csharp
var audioSource = GetComponent();
var smartAgent = GetComponent();
smartAgent.OnAudioResponseReceived += (audioClip) =>
{
audioSource.clip = audioClip;
audioSource.Play();
};
```
**public UnityEvent OnRequestFailed**
- Event that is triggered when a request to the Neocortex project fails.
- Parameters:
- `error`: The error message.
- Example:
```csharp
var smartAgent = GetComponent();
smartAgent.OnRequestFailed += (error) =>
{
Debug.LogError(error);
};
```
### NeocortexAudioReceiver component
The `NeocortexAudioReceiver` component is used to record audio data from the microphone via loudness of the souned, so you can have a hands free chat with the smart agent. On this component you can:
- pick the microphone device to use
- set the amplitude threshold for when to start and stop recording
- set the max wait time for the recording to automatically stop if no sound is detected
**public void StartMicrophone()**
- Starts recording audio from the microphone
- Example:
```csharp
var audioReceiver = GetComponent();
audioReceiver.StartMicrophone();
```
**public void StopMicrophone()**
- Stops recording audio from the microphone
- Example:
```csharp
var audioReceiver = GetComponent();
audioReceiver.StopMicrophone();
```
**public UnityEvent OnAudioRecorded OnAudioRecorded**
- Event that is triggered when audio data is recorded from the microphone.
- Returns:
- `audioClip`: The recorded audio clip.
- Example:
```csharp
var audioReceiver = GetComponent();
audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
{
Debug.Log($"Audio Data Length: {audioClip.samples}");
});
```
## Sample Projects
You can find sample projects that demonstrate how to use the Neocortex Unity SDK in the Package Manager window under the `Samples` section of the Neocortex package.