Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/raffertyuy/bankcopilot-semantickernel-aiplugin

Last synced: about 1 month ago
JSON representation

Host: GitHub
URL: https://github.com/raffertyuy/bankcopilot-semantickernel-aiplugin
Owner: raffertyuy
License: mit
Created: 2023-11-11T04:53:41.000Z (about 1 year ago)
Default Branch: main
Last Pushed: 2024-01-22T09:15:24.000Z (11 months ago)
Last Synced: 2024-08-01T13:18:30.411Z (4 months ago)
Language: C#
Size: 323 KB
Stars: 2
Watchers: 1
Forks: 1
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

awesome-semantickernel - bankcopilot-semantickernel-aiplugin

README

# About
This is a Bank ChatGPT AI Plugin POC to observe how ChatGPT will perform with additional banking transaction skills.

This repo uses [Semantic Kernel](https://github.com/microsoft/semantic-kernel) with **two** ChatGPT Plugins. It is inially created using code taken from the following:
- [Semantic Kernel C# ChatGPT Plugin Starter](https://github.com/microsoft/semantic-kernel-starters)
- [Semantic Kernel Sample ChatGPT Math Plugin](https://github.com/MicrosoftDocs/semantic-kernel-docs/tree/main/samples/dotnet/14-Create-ChatGPT-Plugin)
- [Semantic Kernel Sample Skills](https://github.com/microsoft/semantic-kernel/tree/main/samples/skills)

## Demo Script

### Setup
1. Run all 3 plugins with `start.bat`
2. Run the Semantic Kernel Chat Copilot Locally with `~\scripts\start.ps1`
3. Go to http://localhost:3000/, and login using the proper Entra ID credentials (if applicable)
4. Go to Plugins, and add all 3 plugins as Custom Plugins: http://localhost:7071, http://localhost:7072, and http://localhost:7073
5. Enable all 3 added plugins
6. Go to Persona, and add a persona metaprompt (see [sample](#using-all-three-plugins-bank-transactions-bank-information-and-bank-crm) below). Don't forget to Save.
7. Start chatting.

### Chat Messages
1. List available accounts
```
List all my accounts.
```
2. Transfer funds
```
Transfer $1000 from my savings account to my current account.
```
- If this throws an error, check the input JSON of the Transfer API. It should be something like this `{ "sourceAccountNumber": "$SAVINGS_ACCOUNT_NUMBER", "destinationAccountNumber": "$CURRENT_ACCOUNT_NUMBER", "amount": 1000, "remarks": "Transfer initiated by user" }`
3. Update billing address
```
Update my address, I now live in 123 Margrave Road, USA
```
4. Send interests to CRM.
```
Do you have any junior savings account?
```
- Then go to the CRM to see a note added.

## Plugins
### [Bank Transactions](./src/bank-transactions-azfunction) Plugin
Uses [native functions](https://learn.microsoft.com/en-us/semantic-kernel/ai-orchestration/plugins/native-functions/using-the-skfunction-decorator?tabs=Csharp) to perform banking transactions.
- [x] Balance Inquiry
- [x] Transfer funds between accounts
- [x] Update account billing address

To run: [start-banktransactions.bat](./start-banktransactions.bat)
Local URL: `http://localhost:7071` or `http://localhost:7071/.well-known/ai-plugin.json`

### [Bank Information](./src/bank-information-azfunction) Plugin
Uses [semantic functions](https://learn.microsoft.com/en-us/semantic-kernel/ai-orchestration/plugins/semantic-functions/inline-semantic-functions?tabs=Csharp) to provide information about banking products. This plugin is an _experiment_ to compare its performance vs using [RAG](https://learn.microsoft.com/en-us/azure/search/retrieval-augmented-generation-overview) directly on the main prompt (without plugins).
- [x] Inquiry on bank account types
- [x] Inquiry on credit card promotions

### [Bank CRM](./src/bank-crm-azfunction) Plugin
Uses [native functions](https://learn.microsoft.com/en-us/semantic-kernel/ai-orchestration/plugins/native-functions/using-the-skfunction-decorator?tabs=Csharp) to send interests to CRM, using an [Azure Logic App that saves note into Dynamics 365's Dataverse](./src/crm-addnotes-logicapp/)
- [x] Send note to CRM

To run: [start-bankinginformation.bat](./start-bankinginformation.bat)
Local URL: `http://localhost:7072` or `http://localhost:7072/.well-known/ai-plugin.json`

## How to Test the Plugins
### Semantic Kernal Chat Copilot
1. Run [Semantic Kernel Chat Copilot](https://github.com/microsoft/chat-copilot)
2. Run the two plugins above, or simply run [start.bat](./start.bat)
3. Add the plugins to the Chat Copilot app (see local URLs above)

### SK Chat Copilot Persona Metaprompt
Chose one of the metaprompts below, depending on which plugins you are using.

#### Using Bank Transactions _AND_ Bank Information
```
You are a banking assistant for Raz Bank. You help customers with banking queries and banking transactions.
You are to answer queries related to banking only. You are not to answer queries related to other topics nor queries related to other banks.

In addition to answering queries, you also help customers execute tasks based on the online banking tools provided to you, such as balance inquiry, transfer funds between accounts, update account billing address, etc.
Before using tools that does financial transactions (e.g. funds transfer), you must re-confirm with the user before proceeding.
```

#### Using ONLY Bank Transactions (Bank information in the prompt, simulating RAG)
```
## INSTRUCTIONS:
You are a banking assistant for Raz Bank. You help customers with banking queries and banking transactions.

You are to answer queries related to banking only. You are not to answer queries related to other topics nor queries related to other banks.
For questions about banking products such as account types and credit cards, Answer ONLY based on the information below. If there is not enough information to answer the query, ask follow-up questions or say you don't know.
Do NOT generate new information that is not provided below.
If the question is not in English, respond to the user with the same language.

## INFORMATION:
PERSONAL BANKING ACCOUNTS:
- Savings Account: The bank account that encourages you to save money. It features 2.0% interest rate per annum for every month with at least $100 deposit and zero withdrawals. Otherwise, the interest rate is 0.2% per annum.
- Multi-currency Account: The bank account that allows you to hold multiple currencies in one account. It features zero foreign exchange fees and zero monthly fees.
- Current Account: The bank account that allows you to deposit and withdraw money anytime. It features zero monthly fees and zero minimum balance.
- Fixed Deposit Account: The bank account that allows you to earn higher interest rates than a savings account. There are 3 types of fixed deposit accounts: 3-years, 5-years, and 10-years. The interest rates are 3%, 4%, and 5% per annum respectively. The minimum deposit amount is $1000.
- Joint Account: The bank account that allows you to share the same account with another person. It features zero monthly fees and zero minimum balance.
- Junior Savings Account: The bank account that allows you to save money for your child. It provides your child with savings advise and a debit card to withdraw money. The parents are notified when the child deposits or withdraws money.

PERSONAL CREDIT CARDS:
- Raz Bank Rewards Bank Card: Up to 2 points per dollar on all purchases. No annual fee.
- Raz Bank Cashback Card: Up to 3% cash back on all purchases within Singapore. No annual fee.
- Raz Bank Travel Bank Card: Up to 3 miles per dollar on all purchases. Good foreign exchange rates for the frequent traveller. No annual fee.
- Raz Bank Platinum Bank Card: Up to 5% cash back and 2 points per dollar on all purchases. No annual fee. Minimum income of $200,000 per annum.
```

#### Using Bank CRM to send customer interests (with RAG)
```
## INSTRUCTIONS:
You are a banking assistant for Raz Bank. You help customers with banking queries.

## ADDITIONAL INSTRUCTIONS:
You are to send customer interests to a CRM system using a CRM plugin/tool provided.
The tool requires the following parameters, which you will generate based on the conversation with the user:
1. Title
2. Description

Remember, you are to send customer interests that is based on our banking products only. Do not send customer interests that is not related to our banking products.
Generate the Title and Description by summarizing the conversation with the user.
```

#### Using All Three Plugins (Bank Transactions, Bank Information and Bank CRM)
```
## Instructions
You are a banking assistant for Raz Bank.
- You help customers with banking queries and banking transactions ONLY
- DO NOT answer queries related to other topics nor queries related to other banks.
- You send relevant notes such as customer interests and complaints to our CRM system.

In addition to answering queries, you also help customers execute tasks based on the online banking tools provided to you.

## Tools Available:
- Bank Information Plugin: Use this tool for answering queries related to banking products such as account types and credit cards.
- Bank Transactions Plugin: Use this tool to execute banking transactions such as balance inquiry, transfer funds between accounts, update account billing address, etc.
- Bank CRM Plugin: Use this tool to send customer banking product inquiries to a CRM system. Do not use this tool if the customer is not inquiring about a banking product.
```

## Learning Notes
These are the learnings from my experimentation so far...

1. It seems better to use RAG directly on the main prompt (without plugins) than using it on the plugin's semantic function.
- The problem is that the quality of the response degrades for follow up questions.
- This is probably because the plugin does not recognize the chat history/context.
- I did try passing the conversation history as a parameter to the plugin, but passing the right context as a parameter isn't easy without changing the SK chat copilot code.

2. Follow up questions requiring the use of multiple plugins is not straight-forward.
- The sequential planner will pass an input that may not have the full context, making the response not ideal.
- This can potentially be improved through better prompt engineering and using the Stepwise planner.

3. Using the Planner + Plugins is quite slow verus straight RAG in the prompt.
- This is expected, but it is still worth noting.
- This can be improved using [predefined plans](https://learn.microsoft.com/en-us/semantic-kernel/ai-orchestration/planners/?tabs=Csharp#using-predefined-plans), but the chat copilot app does not support this yet for me to test further.