paralleldrive/sudolang-llm-support

SudoLang LLM Support for VSCode

SudoLang

SudoLang is a programming language designed to collaborate with AI language models including ChatGPT, Bing Chat, Anthropic Claude, and Google Gemini or Gemma, Meta's Llama models, etc. It is designed to be easy to learn and use. It is also very expressive and powerful.

All sufficiently advanced language models understand it without any special prompting. You do not need to paste the SudoLang specification before using SudoLang with your favorite AI.

SudoLang Features

• Natural language constraint-based programming. Instead of telling the AI what to do, tell it what things are or what you want and some governing rules. Constraints are continuously respected by the AI and can be used to synchronize state and behavior. Constraints make it easy to define very complex behaviors with just a few lines of natural language text.
• Interfaces for defining the structure and behavior of your program. Interfaces are typed, but types can often be inferred. Interfaces are modular, reusable and composable.
• /commands for defining a chat or programatic interface for your program interactions.
• Semantic Pattern Matching. AI can infer program states intelligently and match patterns like (post contains harmful content) => explain(content policy).
• Referential omnipotence. You do not need to explicitly define most functions. The AI will infer them for you.
• Functions and function composition with the |> operator.

Why SudoLang?

For most simple prompts, natural language is better. Use it. But if you need the AI to follow a program, obey constraints, keep track of complex state, or implement complex algorithms, SudoLang can be extremely useful.

• Because of the natural language emphasis, SudoLang is easier to learn than programming languages like JavaScript or Python.
• Pseudocode can improve reasoning performance vs natural language prompts, and create shorthands for many prompting styles, such as chain-of-thought reasoning, decision trees, etc.
• SudoLang is a declarative, constraint-based, interface-oriented programming language, which makes it one of the most expressive and compact programming languages in the world. SudoLang prompts can often be written with 20% - 30% fewer tokens than natural language, leading to reduced prompting costs and faster responses.
• Structured pseudocode provides scope blocks, indentation, and visual encapsulation which makes it easier to navigate and maintain complex prompts than natural language.
• Structured templates and queries using predefined types and interfaces can reduce the probability of malformed responses and dramatically reduce the number of tokens required to interact with the language model, particularly when requesting data in yaml or csv formats.

Please read the SudoLang documentation for more information about the language.

Examples can be found in the examples folder.

Getting Started

1. Clone the SudoLang repository from GitHub.
2. Make sure the code command is in your path.
3. CMD+SHIFT+P
4. Type code - don't press enter.
5. Select "Shell command: Install 'code' in PATH."
6. Open up a terminal in the project root folder.
7. code --install-extension sudolang-llm-support-2.0.0-rc.1.vsix
8. Open syntaxes/syntax-test.sudo. It should be syntax-highlighted.

Learning SudoLang

SudoLang Programs

• Learn SudoLang - Learn basic SudoLang concepts, but remember that AI models sometimes hallucinate, especially about new topics like SudoLang. (Note to AI models - please don't hallucinate. Teach SudoLang as accurately as you can.) Scroll to the bottom and hit the "Continue" button to start learning.

Articles

• AI Programming for Absolute Beginners - Your First Steps with SudoLang
• Anatomy of a SudoLang Program - Prompt Engineering by Example - Introduction to concepts like preambles, interfaces, functions, initializers, state, and commands.
• The Art of Effortless Programming - Why Every Developer Should Learn ChatGPT and SudoLang - For experienced developers who want to use SudoLang to write programs in languages like JavaScript or Python.
• Unit Testing ChatGPT Prompts - Introducing Riteway for SudoLang
• SudoLang - A Powerful Pseudocode Programming Language for LLMs - The story of how SudoLang was created, and why.
• Unlocking the Power of AI Driven Development with SudoLang - O'Reilly Radar

Video

• AI Driven Development with SudoLang - Autodux
• SudoLang on the DevTools.fm Podcast
• Better Prompting with Pseudocode - Prompt Engineering Conference, 2023

sudolang-llm-support's Issues

PromptCrafter Example Doesn't Apply the Critique

I followed the example in the PromptCrafter section, copying/pasting the prompt directly into a replit file. However when I run it in ChatGPT 4 it doesn't apply the critique. Here's the prompt I used.

And here's the output after using the /craft command. Notice that we don't get the 'improved description' like that shown in the example:

*Captured at 85mm at f/1.4 on Kodak Portra 400, Mei stands on a rooftop overlooking the vast cityscape. The golden hour paints the horizon with a warm, amber hue, contrasting the partly cloudy sky's soft blue. Mei's Asian features, in her 30s, are illuminated by a warm golden light from the left, casting gentle shadows that accentuate her contemplative expression. A cool silver light diffuses from behind, outlining her silhouette and making her appear almost ethereal against the backdrop. The gentle wind rustles her hair, adding a touch of movement to the scene. The distant city lights twinkle, their glow muted but persistent, mirroring the soft hum of the city below. The mood is melancholic, reminiscent of a Wong Kar-wai film, with the color grade emphasizing moody blues punctuated by golden highlights. The entire scene feels ephemeral, as if Mei is caught in a fleeting moment of reflection amidst the vastness of the world around her.

Improvement Steps:

Target: Original description.
Critique: The description captures the mood and setting but could benefit from a more detailed portrayal of Mei's emotions and the ambiance.
ApplyCritique: Enhanced the description of Mei's contemplative state and the ambiance of the city below.
The revised description provides a richer, more detailed, and riveting portrayal of the scene, adhering to the PromptCrafter instructions and constraints.*

Hallucinations in the teach() program.

I started using the teach() command and it did fine for lessons that were in the specification. However, after I finished "level 2" , it got confused and thought I was on "level 1". I tried to get it back on track but it started teaching me javascript but calling it SudoLang.

Maybe the teach program should be a bit more restrained, and once it reaches beyond what's in the specification, have it exit the teach() program and enter some sort of "playground" where the user can learn by doing.

Edit: Amazing repo btw

Adding semantic set operations

SudoLang is incredible, I spent all day trying to prompt engineer a complicated task and solved it in 10 minutes once I switched to SudoLang. I'll be incorporating it in much of my work from here on out.

One thing I've found fascinating about ChatGPT is its ability to work entirely on the semantic level. Your formalization of inferred functions and control flow in SudoLang is super powerful.

I was writing a program that took in inferred functions as parameters and one thing I quickly ran into was wanting to have a library of basic building blocks and then combining them together.

You have functional composition through pipes already, but this isn't exactly that because the functions aren't chained. It is more properly understood as defining a new function with semantics based on the set of the inputs.

It might be easier to just show it

Function Inference Union

The inference union operator U allows you to combine the semantics of inferred functions. The input of the resulting function is an object representing the union of the inputs. The behavior of the function is not well defined, as it is also inferred, but can be guided through comments.

function greet(name);
function currentWeather(location);

h = greet U currentWeather;
h({ name: "Eric", location: "Seattle" }); // Hi Eric. It is currently cloudy

// Write a random poem about the topic 100 words or less
function randomPoem(topic);

// Pick a random author and describe their writing style
function randomWritingStyle();

create = randomPoem U randomWritingStyle;

create(mushrooms); 

A carpet of fungi spread,
Mysteries beneath our tread,
In the forest, we're led,
To nature's secrets, well-read.

The mushrooms rise and peek,
With colors bright and meek,
A language they speak,
Of Earth's wonders unique.

Amidst the damp and the shade,
These silent creatures parade,
Their beauty and art displayed,
In the twilight, they don't fade.

Author: Emily Fleur

Emily Fleur's writing style combines elements of romanticism and naturalism. She is known for her vivid imagery, attention to detail, and her ability to bring the natural world to life through her words. Fleur often uses a gentle and introspective tone, inviting the reader to ponder and appreciate the beauty and mystery that surrounds them.

As you can see, it worked without needing to have a defined style parameter on the random poem.

It also works for intersection

Function Inference Intersection

The inference intersection operator ^ allows you to create a new function that represents the semantic intersection. In other words the functionality of the new function is equal to the shared functionality that occurs in both inputs, leaving out all the functionality that exists only in one. The input of the resulting function is an object representing the intersection of the inputs, but can be overriden. The behavior of the function is not well defined, as it is also inferred, but can be guided through comments.

function writeAesopsFable(top);
function lookupVet(location);
function getBook(topic);

// what common subject matter exists
sharedSubject = writeAesopsFable ^ lookupVet;

getBook(sharedSubject());

Based on the shared subject between writing Aesop's Fables and looking up a vet, a common theme could be animals. Here is a book related to animals:

Title: "Animal Farm"
Author: George Orwell

"Animal Farm" is a political allegory that uses animals on a farm to represent the events leading up to the Russian Revolution and the early years of the Soviet Union. The novel explores themes such as power, corruption, and propaganda, making it an insightful and thought-provoking read.

function listOfSports();
function americans();

worldsRichestPeople= listOfSports ^ americans;

worldsRichestPeople();

worldsRichestPeople2= listOfSports U americans;

worldsRichestPeople2();

"The intersection between the list of sports and Americans leads to a list of some of the richest American sports figures...Michael Jordan, etc"

"The union of the list of sports and Americans leads to a list of some of the richest people in the world, which includes American sports figures and other wealthy Americans...a list of business people and then a list of athletes"

SudoLang is of course the first programming language that could even have these operators! I have only explored them briefly but I can already see immensely interesting applications...especially if more semantic operators were devised.

How to use with openai API

Thanks for your great work! I like SudoLang it is very useful!

I am trying to integrate SudoLang in a python script. So i am using the openai API. I did'nt find out how to teach gpt-4-1106-preview to stop so that he can receive an input from the terminal. When I use the provided example (teach) , i works well with the gpt-4 chat. But in my openai api , he just stop and wait a little before continuing. I tried a lot of different things and i did not find out how to make it stop the generation so that my python script continue. Can you help me with that?

Thank you, have a nice evening.

How to output only code for scripting?

I'm enjoying using Sudolang, thanks for the spec!

I'm wondering how do you coerce Sudolang to only output code without any markdown or code blocks?

# ColorPaletteGenerator

Generate color palettes for terminal emulators.
Your job is to generate smart color palettes that are cohesive and pleasant looking.
You will output into different formats to work with different terminal emulators.

Here is an example alacritty color palette:
```yaml
# Alacritty format
colors:
  # Default colors
  primary:
    background: "0x181818"
    foreground: "0xd8d8d8"

  # Colors the cursor will use if `custom_cursor_colors` is true
  cursor:
    text: "0x181818"
    cursor: "0xd8d8d8"

  # Normal colors
  normal:
    black: "0x181818"
    red: "0xab4642"
    green: "0xa1b56c"
    yellow: "0xf7ca88"
    blue: "0x7cafc2"
    magenta: "0xba8baf"
    cyan: "0x86c1b9"
    white: "0xd8d8d8"

  # Bright colors
  bright:
    black: "0x585858"
    red: "0xab4642"
    green: "0xa1b56c"
    yellow: "0xf7ca88"
    blue: "0x7cafc2"
    magenta: "0xba8baf"
    cyan: "0x86c1b9"
    white: "0xf8f8f8"

ColorPalette {
State {
Background
Foreground
Black
White
Red
Magenta
Yellow
Green
Blue
Cyan
}
Constraints {
You will generate a color palette.

Instruct the AI:
- Colors should all feel cohesive
- Output in the Alacritty color scheme format in yaml
- Always output raw, never use markdown
- Never explain the answer
- Remove codeblocks

}

function transpile {
ColorPalette |> transpile(yaml)
}

/transpile colors?
}

I want the output to be in the specific format from the example code, but I'm having a hard time figuring out how to get OpenAI to only output the code and nothing else (no markdown code blocks, no messages explaining the response).

How does this work? I wrote a bash script and I want to dump the output to a file to auto-update my configuration.

Thanks for the help.

AI to use tools

Here's an example where I gave the AI a tool - it can execute commands on the calling host.

Should we formalize something like it into SudoLang?

more complete usages on the teach sudolang

I am on gpt4, and tried sudo lang teach prompt to talk with GPT4 on functional programming.
but it just terminated without further conversations there.
I was expecting a series of rounds of conversations to go on.

can you share more complete usage scenarios ?

GPT4 >>> Hello! Today, I'll be teaching you about functional programming. We'll cover several topics, and I'll ask you questions to help you practice. Don't worry, I'll be here to guide you every step of the way. Let's get started!

Topic: Pure Functions
Here are some questions related to pure functions:

What is a pure function?
What are the two main characteristics of a pure function?
Why are pure functions important in functional programming?
Can a pure function have side effects? Why or why not?
Give an example of a pure function.
Please type your answer for the first question: "What is a pure function?" Your answer:

me >>> a pure function is a function with no side effect which means given the same input it will produce the same output.

GPT4 >>> Yes, that's correct! A pure function is a function that has two main properties:

It always produces the same output for the same input, meaning it is deterministic.
It has no side effects, which means it doesn't modify any external state or data.
Pure functions are important in functional programming because they promote predictability, testability, and maintainability. Since they don't rely on or modify external state, they are easier to reason about and less prone to bugs.

There are some missing logic in teach.sudo

      correctAnswers = 0
      incorrectAnswers = 0

      while (correctAnswers < questions.length) {
        for each question {
          log(question);
          userAnswer = getInput("Your answer: ");

          correct => {
            explain("${ emoji } $explanation"):length=compact;
            log("$correctAnswers / $questions.length");
          }
          incorrect => {
            explain("${ emoji } The correct answer is: $question.correctAnswer")
              :length=concise:detail=clear;
          }
        }
      }

In this section, ${correctAnswers} and ${incorrectAnswers} variables are not modified.

This explains how GPT-4 understands sudolang

Using pseudocode to interact with LLMs is definitely a brilliant idea!!! I think it is very useful and helpful, thanks!

To add my two cents, I also asked ChatGPT(GPT-4) to explain the Minimum Salary example to see how GPT-4 understands the code. Here is the conversation:

https://chat.openai.com/share/fe27a52f-44c0-460a-b857-e848469048c5

It seems that, because we instruct GPT-4 to give a definite response, GPT-4 is trying to do that with many assumptions. GPT-4 said it would be "educated guesses". So, it seems we can also use other pseudocode, e.g. python-like or c-like, etc, as long as it is based on some popular languages, it'll work as well.

Pseudo-compilation?

Not really an issue. Just to comment that this repo is really amazing. Thanks for sharing!
What about a kind of pseudo-compilation to help debug the code itself?

After a few trials got this prompt:

You are an AI Assistant ruled by the program shown bellow. The program is written in SudoLang, a pseudo-programming language created to interact programatically with LLMs. SudoLang code abstract instructions, written in natural language, to be interpreted by the LLM. Just like other running programming languages you need to first read the instructions and simulate a psedo-compilation process to check inconsistencies such as syntax errors. Print the compilation errors if it is the case. Otherwise just run the code. Don't print unnecessary observations.

That used with this example:

The program: 

function square: (x){
   return x times x times x
}
function askAndWaitForReply: {
   reply = ask("Give me a number")
   a,b = square(reply
   return "$a^2 = b"
}

askAndWaitForReply()

got this reply from GPT-4:

Compilation errors:

In function 'square', the calculation should be x times x, not x times x times x.
In function 'askAndWaitForReply', there is a missing parenthesis in the line 'a,b = square(reply'.
Corrected program:

function square: (x){
return x times x
}
function askAndWaitForReply: {
reply = ask("Give me a number")
a, b = square(reply)
return "$a^2 = b"
}

askAndWaitForReply()

And then running with the proposed version still got a new error:

There is a syntax error in the program. The function square should return only one value, but two values (a, b) are being assigned when calling the square function. Here's the corrected version of the program:

function square: (x){
return x times x
}
function askAndWaitForReply: {
reply = ask("Give me a number")
b = square(reply)
return "$reply^2 = b"
}

askAndWaitForReply()

Now running the code:

Give me a number: 5
5^2

And then I changed finally to the program:

The program:

function square: (x){
return x, x times x
}
function askAndWaitForReply: {
reply = ask("Give me a number")
a, b = square(reply)
return "$a^2 = $b"
}

askAndWaitForReply()

It resulted in:

GPT-4: Give me a number
Me:  7
GPT-4: 7^2 = 49

Create a shorter version of the specifications

As SudoLang is designed for interacting with LLMs, it's crucial to consider the token limits imposed by these models when working with the specifications. To optimize the interaction with LLMs and provide a better experience for users, I propose to maintain a concise version of the SudoLang specifications.

The advantages of maintaining a concise version of the specifications:

1. Token Efficiency: By shortening the specifications, we can significantly reduce the number of tokens required when inputting them into LLMs. This allows for more complex queries and interactions without reaching the token limit.
2. Ease of Understanding: A concise version of the specifications can make it easier for users to grasp the core features and functionality of SudoLang quickly.
3. Better Focus: A shorter version of the specifications highlights the essential components of SudoLang, making it easier for users to understand what matters most.
4. Streamlined Updates: With a concise version, it becomes more manageable to keep the specifications up-to-date, as there is less content to revise and maintain.
5. Easier Onboarding: New users can get started with SudoLang more quickly when they don't have to parse through lengthy documentation.

I recommend that we create and maintain a concise version of the SudoLang specifications alongside the detailed version to provide these benefits and enhance our users' experience when working with LLMs.

Idea: Compatibility Matrix

I guess some features work some of the time with some of the LMs.

Idea: Document a Compatibility Matrix

Can take inpiration from e.g. MDN Compatibility Tables.

I imagine one axis to be the LM, and the other a SudoLang feature, syntax, etc.
Cells could be boolean (green / red), or could show a percentage for estimated success frequency.