Determining Script types with Decision Models

Determining Script types with Decision Models

Today, I am learning about the newest type of models, System One models which have the ability to produce typed decisions and probabilities. If you have heard of Jev - this is a type of System One model. These models are trained to make decisions and provide a simple output rather than full text output with reasoning around the decision, the added benefit they are blazingly fast in the cloud service.

If you are interested in learning more about the type of models, check out the TypeSafe.ai documentation to understand the concepts: System One Concepts | TypeSafe Docs

For my learning, I am using local models for decision making using Ollama and the models that are currently available (5 at the time of writing):

There are 3 supported models of running a decision models:

For my research, I am using types of scripts in PnP Script samples to categorise them according to their type as an example e.g., which ones are PnP PowerShell, CLI for Microsoft 365 and Microsoft Graph SDK as experiment - as my use case (lol, this may not be the optimal use case for this model).

This was setup in Python with the following code, using 4 example txt files containing 4 types of scripts that contain:

• PnP PowerShell script
• CLI for Microsoft 365 Script
• Microsoft Graph PowerShell SDK based script
• PowerApps PowerShell script (representing the negative test)

Whilst this is a very small sample, the purpose is to determine if the decision models contain knowledge of the PowerShell modules.

import requests
import json
from pathlib import Path

input_files = ["Example1.txt", "Example2.txt", "Example3.txt", "Example4.txt"]

for filename in input_files:
    print(f"Processing file: {filename}")

    file_content = Path(filename).read_text(encoding="utf-8")

    payload = {
        "model": "clef:27b-nvfp4", #"tev1", "nimble", "clef-flash"
        "state": file_content,
        "questions": {
            "label": {
                "type": "choice",
                "instructions": "What type of script is this?",
                "criteria": {
                    "pnp-powershell": "PnP PowerShell Script which can use Connect-PnPOnline connection",
                    "m365-cli": "CLI for Microsoft 365 script which can use m365 connection",
                    "msgraph": "Microsoft Graph PowerShell SDK Script which can use Connect-MgGraph connection",
                },
            },
        },
    }

    response = requests.post(
        "http://localhost:11434/v1/systemone",
        json=payload,
        timeout=120,
    )
    
    response.raise_for_status()

    result = response.json()
    #print(json.dumps(result, indent=2, ensure_ascii=False))
    print(f"File: {filename} - Choice: {result['answers']['label']['choice']}  with confidence {result['answers']['label']['confidence']}")  # "technical"
    

I’m not using any SDK here, there is an SDK available on TypeSafe website

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
from pathlib import Path

client = TypeSafeClient(
    base_url="http://localhost:11434",
    api_key="no_api_key_required",
)

input_files = ["Example1.txt", "Example2.txt", "Example3.txt", "Example4.txt"]

for filename in input_files:
    print(f"Processing file: {filename}")

    file_content = Path(filename).read_text(encoding="utf-8")

    response = client.system_one(
        model="clef:27b-nvfp4", #"tev1", "nimble", "clef-flash"
        state=file_content,
        questions={
            "script": Choice(
                instructions="What type of script is this?",
                criteria={
                    "pnp-powershell": "PnP PowerShell Script which can use Connect-PnPOnline connection",
                    "m365-cli": "CLI for Microsoft 365 script which can use m365 connection",
                    "msgraph": "Microsoft Graph PowerShell SDK Script which can use Connect-MgGraph connection",
                },
            )
        },
    )

    print(f"File: {filename} - Choice: {response.answers['script'].choice}  with confidence {response.answers['script'].confidence:.1%}")  # "technical"
    

The result is (for the SDK run, trimmed for brevity):

Using model: clef:27b-nvfp4
File: Example1.txt - Choice: pnp-powershell  with confidence 96.5%
File: Example2.txt - Choice: m365-cli  with confidence 89.8%
File: Example3.txt - Choice: msgraph  with confidence 96.3%
File: Example4.txt - Choice: msgraph  with confidence 11.1%

Using model: tev1
File: Example1.txt - Choice: pnp-powershell  with confidence 98.4%
File: Example2.txt - Choice: m365-cli  with confidence 96.7%
File: Example3.txt - Choice: msgraph  with confidence 98.9%
File: Example4.txt - Choice: msgraph  with confidence 56.1%

Using model: nimble
File: Example1.txt - Choice: pnp-powershell  with confidence 97.2%
File: Example2.txt - Choice: m365-cli  with confidence 94.7%
File: Example3.txt - Choice: msgraph  with confidence 99.7%
File: Example4.txt - Choice: msgraph  with confidence 34.9%

Using model: clef-flash
File: Example1.txt - Choice: pnp-powershell  with confidence 93.1%
File: Example2.txt - Choice: m365-cli  with confidence 72.9%
File: Example3.txt - Choice: msgraph  with confidence 90.6%
File: Example4.txt - Choice: m365-cli  with confidence 53.5%

Here are the results (using an expanded script):

I have omitted Laya model (“laya:421m-typed-decisions-mlx-fp16 “) from these experiments as scripts exceeded the context window. Laya is a small model and this I feel is the wrong use case.

Ooo Microsoft Decision 1 Model

Microsoft have their own decision model called Microsoft-Decision-1 Model that would be good to compare against - however this is a cloud model so will have a huge jump in speed.

So adapting the script to the following, we don’t need to loop through the models but we do need to specify the service endpoint, which I have setup in an .env file. The other key service you need to configure, is in Microsoft Foundry, you will need to deploy a Decision Model - in my instance, I provision from the catalog “Microsoft-Decision-1” and go with the default name called “Decision-1”.

import dotenv
import os
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
from pathlib import Path
from IPython.display import Markdown, display

dotenv.load_dotenv()

assert os.getenv("AZURE_ENDPOINT"), "AZURE_ENDPOINT is not set"
assert os.getenv("AZURE_API_KEY"), "AZURE_API_KEY is not set"

client = TypeSafeClient(
    base_url=os.getenv("AZURE_ENDPOINT"),
    api_key=os.getenv("AZURE_API_KEY"),
)

input_files = ["Example1.txt", "Example2.txt", "Example3.txt", "Example4.txt"]
results = []

for filename in input_files:
    print(f"Processing file: {filename}")

    file_content = Path(filename).read_text(encoding="utf-8")

    response = client.system_one(
        model="Decision-1",
        state=file_content,
        questions={
            "script": Choice(
                instructions="What type of script is this?",
                criteria={
                    "pnp-powershell": "PnP PowerShell Script which can use Connect-PnPOnline connection",
                    "m365-cli": "CLI for Microsoft 365 script which can use m365 connection",
                    "msgraph": "Microsoft Graph PowerShell SDK Script which can use Connect-MgGraph connection",
                },
            )
        },
    )

    answer = response.answers['script']
    results.append({
        "model": "Microsoft-Decision-1",
        "file": filename,
        "choice": answer.choice,
        "confidence": answer.confidence,
    })
    print(f"File: {filename} - Choice: {answer.choice}  with confidence {answer.confidence:.1%}")

table_rows = "\n".join(
    "| " + " | ".join(
        str(value).replace("|", "|").replace("\n", " ").replace("\r", " ")
        for value in (row['model'], row['file'], row['choice'], f"{row['confidence']:.1%}")
    ) + " |"
    for row in results
)
markdown_table = (
    "| Model | File | Choice | Confidence |\n"
    "| --- | --- | --- | --- |\n"
    + table_rows
)
display(Markdown(markdown_table))

Which when run (and of course is substantially faster being a cloud model…)

Here are the results:

File: Example1.txt - Choice: pnp-powershell  with confidence 99.8%
File: Example2.txt - Choice: m365-cli  with confidence 99.8%
File: Example3.txt - Choice: msgraph  with confidence 99.7%
File: Example4.txt - Choice: m365-cli  with confidence 57.8%

One observation is that the model accuracy was higher with the Microsoft-Decison-1 model but I felt that the other local models whilst slower to run did a pretty decent job. This is a small dataset though, to make a firm judgement.

Wrapping up

Yes, I was totally playing with the Script use case, and probably isn’t the most optimal - you could achieve this is a deterministic way, but it did give a scenario to focus on and further understand the potential of System One models - I am looking at potentially looking at this as a way to determine the type of scripts in Script samples for an upcoming extraction to separate files and whether this would support that scenario in the determination of what module was used.

Interesting, learning outside the model capabilities, like with other tools, or agents, good descriptions are definitely needed to improve confidence and matching scores.

Useful References

• Deploy and use Microsoft-Decision-1 in Microsoft Foundry | Microsoft Learn
• Introducing Microsoft-Decision-1, our model for fast decision-making | Commandline
• TypeSafe Python SDK

Enjoy!