pen-lang/pen

The parallel, concurrent, and functional programming language for scalable software development

Pen programming language

Pen is the parallel, concurrent, and functional programming language focused on application programming following Go's philosophy. It aims for further simplicity, testability, and portability to empower team (v. individual) and/or long-term (v. short-term) productivity.

Its syntax, type system, effect system, and module system are fashioned to achieve those goals being simple and easy to grasp for both newcomers and experts. One of the biggest differences from the other functional languages is polymorphism without generics.

Pen provides the two built-in functions of go and race to represent many concurrent/parallel computation patterns. Thanks to its syntax, type system, and the state-of-the-art reference counting garbage collection, programs are always memory safe and data-race free.

System libraries and runtime in Pen are detachable from applications. Thanks to this, Pen can compile the same applications even for WebAssembly and WASI. Pen also provides Rust/C FFI to reuse existing libraries written in those languages.

import Core'Number
import Os'File

# The `\` prefix for λ denotes a function.
findAnswer = \(kind string) number {
  # Secret source...

  21
}

main = \(ctx context) none {
  # The `go` function runs a given function in parallel.
  # `x` is a future for the computed value.
  x = go(\() number { findAnswer("humanity") })
  y = findAnswer("dolphins")

  _ = File'Write(ctx, File'StdOut(), Number'String(x() + y))

  none
}

Install

Pen is available via Homebrew.

brew install pen-lang/pen/pen

For more information, see Install.

Examples

See the examples directory.

Documentation

• Getting started
• Guides
  • Building an executable
  • Creating a library
  • Using a library
• Language reference
  • Syntax
  • Types

Comparison with Go

Overview

	Pen	Go
Domain	Application programming	System programming
Paradigm	Functional	Imperative / object-oriented
Memory management	Reference counting	Concurrent mark-and-sweep
System library	Your choice!	Built-in
Values	Immutable	Mutable

Runtime

	Pen	Go
Context switch	Continuations	Platform dependent
Concurrent computation	Built-in functions	go expression
Synchronization	Futures, lazy lists	Channels, concurrent data structures
Data race prevention	Built into GC	Dynamic analysis
Resource management	Built into GC	defer statement
Error handling	error type, ? operator	error type, multi-value return
Exception	None	panic and recover functions

Types

	Pen	Go
Number	number (IEEE 754)	int, float64, ...
Sequence	[number] (lazy list)	[]int (array or slice)
Map	{string: number}	map[string]int
Optional value	none, union types	null pointer (or zero value)
Function	\(number, boolean) string	func(int, bool) string
Union	number | string	Interface
Top type	any	any (interface{})
Interface	Records	Interface
Futures	Functions (thunks)	None
Concurrent queue	[number], built-in functions	chan int

The \ (lambda, λ) notation in function types and literals originates from other functional programming languages like Haskell.

Technical design

Polymorphism without generics

Pen explicitly omit generics (or specifically parametric polymorphism for user-defined functions and types) from its language features as well as the original Go. It is one of the biggest experiments in the language as most of existing functional languages have generics as their primary features.

Instead, we explore polymorphism with other language features, such as generic constructs (e.g. list comprehension and pattern matches,) subtyping, top types, reflection, code generation, and so on. A belief behind this decision is that Pen can achieve the same flexibility as other languages reducing complexity of the language itself. For the same reason, we don't adopt macros as we believe they are too powerful for humanity to handle.

Dynamic effect system

Pen does not adopt any formal effect system of algebraic effects or monads. Instead, Pen rather uses a simple rule to manage side effects: all effects are passed down from the main functions to child functions. So unless we pass those impure functions to other functions explicitly, they are always pure. As such, Pen is an impure functional programming language although all runtime values are immutable. However, it still provides many of the same benefits purely functional languages do, such as determinicity and testability.

The reason we do not adopt any formal and statically provable effect system is to keep the language and its type system simple and lean for the purpose of improving developer productivity and software development scalability; we want to make Pen accessible and easy to learn for both newbie and expert programmers.

Context switch

Like Go, every function in Pen is suspendable and can be called asynchronously. This is realized by intermediate representation compiled into Continuation Passing Style (CPS) which also enables proper tail calls. Thus, Pen implements context switch without any platform-dependent codes for slight sacrifice of performance while Go requires logic written in assembly languages.

Currently, Pen does not use delimited continuations for the following reasons.

• Traditional continuations are sufficient for our use cases, such as asynchronous programming.
• Delimited continuations require heap allocations although the second-class continuations do not.

Reference counting GC

Pen implements the Perceus reference counting as its GC. Thanks to the state-of-the-art ownership-based RC algorithm, programs written in Pen performs much less than traditional RC where every data transfer or mutation requires counting operations. In addition, the algorithm reduces heap allocations significantly for records behind unique references, which brings practical performance without introducing unsafe mutability.

See also How to Implement the Perceus Reference Counting Garbage Collection.

Inductive values

TBD

Stackful coroutines

TBD

Contributing

Pen is under heavy development. Feel free to post Issues and Discussions!

Workflows

Installing from source

See Install.

Building crates

tools/build.sh

Running unit tests

tools/unit_test.sh

Running integration tests

tools/build.sh
tools/integration_test.sh

Running benchmarks

Those benchmarks include ones written in both Pen and Rust.

tools/benchmark.sh

Linting crates

tools/lint.sh

Formatting crates

tools/format.sh

Directory structure

• cmd: Commands
  • pen: pen command
• lib: Libraries for compiler, formatter, documentation generator, etc.
  • app: Platform-agnostic application logic for pen command
  • infra: Platform-dependent logic for pen command
  • ast: Abstract Syntax Tree (AST) types
  • hir: High-level Intermediate Representation (HIR) types and semantics
  • mir: Mid-level Intermediate Representation (MIR)
  • ast-hir: AST to HIR compiler
  • hir-mir: HIR to MIR compiler
  • mir-fmm: MIR to F-- compiler
• packages: Packages written in Pen
  • core: Package for platform-independent algorithms and data structures
  • os: Package for a common OS interface
• tools: Developer and CI tools
• doc: Documentation at pen-lang.org

License

Pen is dual-licensed under MIT and Apache 2.0.

pen's Issues

File'Move function in Os package

• Add a function to move files and directories in the Os package.

Test summary

Problems

• It's difficult to figure out how much portion of tests failed on pen test if test suites are large.
• pen test command outputs no message if no test file is defined.
  • It is difficult for users to know if it's expected or not.
• Related to #441.

Solution

• Output test summary after tests run.

Thunk environment not droppped when thunk is forced

• Thunk environments are not dropped properly when the thunks are forced.
  • If it's not forced, environments seem to be dropped properly.
  • Example: https://github.com/raviqqe/pen-examples/blob/main/string-join-leak/Main.pen

Restrict field access of private records

Problem

• Currently, fields of private records are accessible as long as they are open.
• That should be prohibited because they are private.

Solution

• Prevent access of fields of private records outside modules where they are defined.

Document build dependencies on building from source

• Document dependencies on building from source.
• List up all dependencies which are automatically installed when you use Homebrew.

TCP/UDP interface in Os package

• Add TCP/UDP interface in the Os package

Efficient C calling convention in FFI

Problems

• Foreign Function Interface (FFI) doesn't support the full C calling convention.
  • Especially around multi-word values (e.g. any and union types) and unboxed records which are always boxed currently
• This is inconvenient as developers need to manually box them in record types.

Examples

• NormalFile record in the Os package

Solution

• Support the full C calling convention in FFI.
  • Simply specifying the C calling convention in foreign export/import makes function signatures compatible with C.

References

• MLIR language reference
• C wrapper emission | MLIR

Disallow union types in FFI

• Currently, union types are not supported as they can't be interpreted directly in foreign languages.
• Invalidate use of union types in export/import syntax of FFI.

Debug function

Problem

• It is hard to debug programs without looking at their intermediate states at runtime.
• Currently, we don't have such a way to peek those states.

Proposed solution

• Implement a debug built-in function which pretty-prints any runtime values to a console.
• It should print those values only if a DEBUG environment variable is set so that we don't break an effect system.

debug = \(x any) {
  ...
}

Immediate closure dropping on function calls

Problems

• The current implementation of closures in MIR is unnecessarily complex because it's imitation of other languages like OCaml which allow recursive closure definitions although Pen doesn't.

Solution

• We should be able to drop or move closures extracting environment variables inside immediately at the beginning of function calls.
  • This is because we cannot define any recursive closure.
  • Let-recursive expressions need to replaced with let-function expressions that are not recursive.

References

• #294

None type in MIR

Syntax highlight in documentation

• Highlight codes in Pen in the mdbook documentation.
  • Use Prism.js?

`test-main` crate separated from `Test` package

Problem

• Users need to add the Test package to their packages' configuration although it's essentially not necessary.

Solution

• Move out pen-link.sh and a ffi crate in the Test package into a test-main crate.
  • The new crate can be under a cmd directory?
• Update documentation accordingly.

Code generator

Problem

• Currently, Pen has no method to do metaprogramming.
• Use cases
  • Stringify
  • Serialization and deserialization
    • e.g. JSON, S-expression, and Protocol Buffer

Solution

• As an MVP of metaprogramming, implement code generator.
  • Its design would be similar to go generate.
  • https://go.dev/blog/generate

JSON library package

Problem

• It's difficult to serialize or deserialize data from and into JSON.

Solution

• Provide a standard package of JSON serialization/deserialization.

Type check for try operator in functions with non-result return types

Separate object directories for different targets

• Separate object directories for different target triples.
  • The default target triple's can be named default as we can't know its name in the app crate currently.

Invalidate use of non-existent types in type definitions and aliases

• In the current implementation, we do not check the existence of reference and record types in type definitions or aliases.
  • Because type check there can be skipped if the types are not used anywhere.
• Check them to catch errors properly.

Fix wasm32-wasi cross compile

• There seems to be ABI incompatibility between codes generated by Pen and Rust.

Code formatter

Problem

• Currently, developers need to format source files.
• That is not only tedious toil but also leads to inconsistency of the coding styles.

Solution

• Provide a default code formatter.

Record field access restriction

• Invalidate private field access of records outside modules where they are defined.
  • Both in construction and deconstruction of records

`Flag` library package

Problems

• There is no quick and simple way to pass flags to applications through their command line arguments.
• Using command line arguments as string lists is pretty hard and cumbersome.

Solution

• Add Flag package.
• It provides utility functions to parse command line arguments and converts them into flag values.
• Its features can be as simple as the flag package in Go.

FFI cross compilation for application packages

• Currently, FFI cross compilation is not fully supported for application packages.
• Delegate definitions of application linkers to system packages?

pen/lib/infra/src/application_linker.rs

Line 16 in 1f5d8cc

impl app::infra::ApplicationLinker for ApplicationLinker {

Comprehensive error messages of module imports

Problems

• Currently, module import errors are difficult to be interpreted.
  • Because the build system simply pass through errors from Ninja on missing interface files.

Solution

• Detect those errors before actually we run the ninja command on build.
  • Maybe on pen resolve-dependencies?
• Emit human-friendly error messages when imported modules are missing.

Use C calling convention for exported test functions

Problems

• Currently, test functions in test modules are exported with the calling convention of Pen itself.
• That incurs extra complexity to the codes running those functions that are currently written in Rust.

Solution

• Use the C calling convention for exported test functions instead.

References

• Blocked by #246.

Can not build on M1 Mac.

Hi, this looks a nice project!
I installed pen from homebrew on M1 Mac.
But failed to build the newly created project.
It seems that target-triple which is put for rustc by pen is not valid.
At least in my environment, rustc can accept aarch64-apple-darwin, not arm64-apple-darwin.

❯ pen create foo
❯ cd foo   
❯ pen build 
[1/20] compiling module of /Users/monochrome/foo/.pen/packages/d3ac9f08bee9d692/String.pen
[2/20] resolving dependency of /Users/monochrome/foo/Main.pen
[3/20] resolving dependency of /Users/monochrome/foo/Main.pen
[4/20] compiling module of /Users/monochrome/foo/.pen/packages/d3ac9f08bee9d692/Error.pen
[5/20] resolving dependency of /Users/monochrome/foo/.pen/packages/c157c35e8d26a22f/MainFunction.pen
[6/20] resolving dependency of /Users/monochrome/foo/.pen/packages/c157c35e8d26a22f/MainFunction.pen
[7/20] resolving dependency of /Users/monochrome/foo/.pen/packages/c157c35e8d26a22f/Os.pen
[8/20] resolving dependency of /Users/monochrome/foo/.pen/packages/c157c35e8d26a22f/Os.pen
[9/20] generating object file for /Users/monochrome/foo/.pen/packages/d3ac9f08bee9d692/String.pen
[10/20] generating object file for /Users/monochrome/foo/.pen/packages/d3ac9f08bee9d692/Error.pen
[11/20] compiling module of /Users/monochrome/foo/.pen/packages/d3ac9f08bee9d692/List.pen
[12/20] generating object file for /Users/monochrome/foo/.pen/packages/d3ac9f08bee9d692/List.pen
[13/20] compiling module of /Users/monochrome/foo/.pen/packages/c157c35e8d26a22f/Os.pen
[14/20] compiling module of /Users/monochrome/foo/.pen/packages/c157c35e8d26a22f/MainFunction.pen
[15/20] generating object file for /Users/monochrome/foo/.pen/packages/c157c35e8d26a22f/MainFunction.pen
[16/20] compiling module of /Users/monochrome/foo/Main.pen
[17/20] /Users/monochrome/foo/.pen/packages/d3ac9f08bee9d692/pen-ffi.sh -t arm64-apple-darwin /Users/monochrome/foo/.pen/objects/d3ac9f08bee9d692.a
x1b[0;31merrorx1b[0m: /Users/monochrome/foo/.pen/objects/d3ac9f08bee9d692.a 
/Users/monochrome/foo/.pen/packages/d3ac9f08bee9d692/pen-ffi.sh -t arm64-apple-darwin /Users/monochrome/foo/.pen/objects/d3ac9f08bee9d692.a
x1b[0;31merrorx1b[0m: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `rustc - --crate-name ___ --print=file-names --target arm64-apple-darwin --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit status: 1)
  --- stderr
  error: Error loading target specification: Could not find specification for target "arm64-apple-darwin". Run `rustc --print target-list` for a list of built-in targets

[18/20] generating object file for /Users/monochrome/GitHub/foo/Main.pen
[19/20] generating object file for /Users/monochrome/GitHub/foo/.pen/packages/c157c35e8d26a22f/Os.pen
[20/20] /Users/monochrome/GitHub/foo/.pen/packages/c157c35e8d26a22f/pen-ffi.sh -t arm64-apple-darwin /Users/monochrome/GitHub/foo/.pen/objects/c157c35e8d26a22f.a
x1b[0;31merrorx1b[0m: /Users/monochrome/GitHub/foo/.pen/objects/c157c35e8d26a22f.a 
/Users/monochrome/GitHub/foo/.pen/packages/c157c35e8d26a22f/pen-ffi.sh -t arm64-apple-darwin /Users/monochrome/GitHub/foo/.pen/objects/c157c35e8d26a22f.a
x1b[0;31merrorx1b[0m: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `rustc - --crate-name ___ --print=file-names --target arm64-apple-darwin --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit status: 1)
  --- stderr
  error: Error loading target specification: Could not find specification for target "arm64-apple-darwin". Run `rustc --print target-list` for a list of built-in targets

error: command exited with status code 1

Enhanced namespace management with package and module systems

• Related to #204

aarch64-apple-darwin: binary killed on Apple silicon due to policy violation

I modified the hello world program, as generated by pen create, to read as follows:

import System'Context { Context }
import System'File

type redblacknode {
  red boolean
}

main = \(ctx Context) number {
  File'Write(ctx, File'StdOut(), "Hello, world!\n")

  0
}

Attempting to build and run this program results in a binary that immediately crashes:

~/code/pen-trees 〉pen build && ./app
[6/6] /Users/dylhunn/code/pen-trees/.p...ult/archives/lib7e2c4ee33a26eae0_ffi.a
zsh: killed     ./app

Removing the record type causes the program to build and run as expected, printing "Hello, world!".

I am on an Apple Silicon Macbook Pro, running MacOS 11.4. I am using pen 0.2.1 as installed via brew.

I'm unable to even load the binary in lldb (note that I'm just trying to load lldb, not even run the target):

~/code/pen-trees 〉lldb ./app 
(lldb) target create "./app"
zsh: killed     lldb ./app

Reference counting for custom FFI data types

• Allow FFI libraries to define their own data types and collect garbage of them through reference counting.
• We can simply use variant types in MIR for those custom data types.

Tutorial documentation

Expected contents

• Writing expressions
• Adding modules
• Managing side effects
• Creating library packages

Clone function compatibility with Rust in type information

Problems

• The current implementation in MIR has a clone function signature of fn clone(payload: u64) in type information global variables.
• That prevents us from using clone functions in Rust in FFI that return totally new objects.
  • e.g. Box

Solution

• Change the signature to fn clone(payload: u64) -> u64.

File system API in OS standard package

• Blocked by #142.

`pen test` fails with no test file emitting incoprehensive error message

Problems

• pen test fails with no test file emitting incomprehensive error message.
• That is not user friendly.

Solution

• pen test command should success with no test file.
• If there is no test file, it should output a special message like no test defined.

Improved parse errors

• Currently, syntax parse errors from the compiler are very hard to interpret.
• This is because we use utility combinators from the attempt.rs module which prevents the commitment of parsers in most cases.
  • Although the combinators prevent hard-to-debug parser definition bugs because parser states are always rolled back on failures.
• Refactor parser combinators using attempt combinators only when necessary while removing the attempt.rs module.

Deterministic testing framework

• pen test subcommand runs all tests in a package.
• Files named *.test.pen are test files.
• Functions named Test* are all test functions.
  • Their arguments are context types which handle mutable states.

Improved build error messages

• Improve build error messages by making them more comprehensive to new users.
• Is it possible to hide the use of the ninja command from the message?

References

• ninja-build/ninja#480 (reference)

Running multiple tests fail on macOS x86-64 target

Problem

• Running multiple tests on macOS x86-64 target fails.
  • All tests fail with FAIL status while they succeed on Linux.
• Running a single test on the same target succeeds.
• Example on GitHub Actions

Solution

• Fix it.

Zero-argument top-level functions as thunks

Modulo function in Core package

• Add a Modulo function to calculate modulo in the Core package

Refactor package and module resolution on building packages

Problems

• External package resolution and enumeration to build main packages
• Module resolution and enumeration to build main packages
  • Use the list of modules to produce archive files per packages including the main packages themselves.

Solution

• Build archive files for each package.
• Build a dependency graph of external packages using some graph algorithm.
  • petgraph

Asynchronous operations

Problems

• All I/O are blocking in the current implementation.
• That leads to low usage of CPU.

Solution

Implement MVP of asynchronous operations.

Scheduler

• For MVP, the implementation can be cooperative and polling-based.
• It iterates over coroutines not resolved yet in a queue and tries resolving them one by one.

Coroutines

• Coroutines return boolean values that indicate if they have been resolved or not.
• If the values are false, put them into a queue of unresolved coroutines and try resolving the next one.

Use relative paths instead of absolute paths for module prefixes

• It's a security issue.

Omit function drops on function calls

Remove use of weak linkage for type information in MIR

• We need to introduce type declarations in MIR.

Document function expressions

Private modules

Problems

• All modules in packages are exposed to other packages.
• We often want to hide those implementation details of packages.

Solution

• Allow modules to be private in packages.
  • Private modules are only accessible inside packages where they reside.
• Use capitalization for making modules public or private.

Stream-like list type

• Make a list type a stream type like the one in Scala.
  • Evaluate the first and rest elements lazily.
• Disallow use of ? operators in list literals.
• We need to change the syntax of the if-list expression as below to evaluate elements lazily.

if [x, ...xs] = xs {
  [x(), ...xs]
} else {
  []
}

C calling convention in FFI

• raviqqe/fmm#186

Improved parse error for suffix operators

• Fix backtracking in suffix operator parsers to fix positions in error messages.

  • pen/lib/parse/src/parsers.rs

    Line 410 in 7301545

    // TODO Move positions into suffix operators.

Full code examples

Problems

Developers who are new to the language can't grasp how the whole applications written in it would look like.

Solution

Implement full code examples of applications.

Candidates

Simple applications

• Hello, world!
• echo
• cat
• ls
• yes

Complex applications

• HTTP server
• Web scraper
• Life game