Giter Site home page Giter Site logo

linguini's Introduction

.NET Nuget Nuget Nuget Nuget

Linguini

Linguini is a C# implementation of Project Fluent, a localization system for natural-sounding translations with features like:

Asymmetric Localization

Natural-sounding translations with genders and grammatical cases only when necessary. Expressiveness is not limited by the grammar of the source language.

Progressive Enhancement

Translations are isolated; locale-specific logic doesn't leak to other locales. Authors can iteratively improve translations without impact on other languages.

Modular

Linguini is highly modular. You only can use the parts you need. Need just parsing? Get Linguni.Syntax. Need only Plural Rules data? Get PluralRules.Generator and connect to XML CLDR Plural rules data.

Performant

Linguini uses a zero-copy parser to parse the resources. While at the moment, there are no benchmarks, it is used by RobustToolbox as a localization framework.

How to get it?

To install the Fluent Bundle type in your console:

dotnet add package Linguini.Bundle

You can also follow other NuGet installation instructions. E.g. :

paket add Linguini.Bundle

Or copy this code to your PackageReference

<PackageReference Include="Linguini.Bundle" Version="0.7.0" />

How to use it?

For a 2-minute tour of Linguini, add this to your C# code:

var bundler = LinguiniBuilder.Builder()
    .CultureInfo(new CultureInfo("en"))
    .AddResource("hello-user =  Hello, { $username }!")
    .UncheckedBuild();

var message = bundler.GetAttrMessage("hello-user",  ("username", (FluentString)"Test"));
Assert.AreEqual("Hello, Test!", message);

The 10 min tour - What do the lines mean?

Let's go line by line and see how LinguiniBuilder works.

  1. Init - This creates a LinguiniBuilder using a type-safe builder pattern.

    var bundler = LinguiniBuilder.Builder()
  2. Set the language - a translation bundle must have a language to translate to. In this case, we choose the English language.

    .CultureInfo(new CultureInfo("en"))
  3. Add a resource - a translation bundle without resources is pointless. We choose inline string for ease of the example.

        .AddResource("hello-user =  Hello, { $username }!")

    If you need an example of passing files, here is a oneliner example assuming you defined using var streamReader = new StreamReader(path_to_file); somewhere (Hint: no need to use StreamReader, any TextReader will do).

         .AddResource(streamReader)
  4. Complete the ResourceBundle - We call UncheckedBuild() to convert a builder to a bundle. The bundle will parse its resources and report errors. Since we don't care about errors, we are fine with ResourceBundle throwing errors.

        .UncheckedBuild();
    
  5. Get hello-user term where username is Test.

    bundler.GetAttrMessage("hello-user",  ("username", (FluentString)"Test"));

Quick questions and answers

Why FluentBundle isn't thread-safe?

Making it concurrent could add a performance penalty; otherwise, a concurrent bundle would be the default. To make it thread-safe, add UseConcurrent() in builder:

    var bundler = LinguiniBuilder.Builder()
       .CultureInfo(new CultureInfo("en"))
       .AddResource("hello-user =  Hello, { $username }!")
       .UseConcurrent()
       .UncheckedBuild();

Or set UseConcurrent = true in FluentBundleOption passed to the factory method:

    var x = new FluentBundleOption()
    {
        UseConcurrent = true,
    };
    var bundle = FluentBundle.MakeUnchecked(defaultBundleOpt);

Isn't .ftl FreeMarker Template language?

No, it can also be Fluent, a localization system by Mozilla.

What syntax does Fluent Localization use?

For more details, see Fluent syntax guide.

Licenses

Linguini Bundle and associated projects are licensed under Apache and MIT licenses. Consult the LICENSE-MIT and LICENSE-APACHE for more detail.

linguini's People

Contributors

abcdefg30 avatar electrojr avatar mailaender avatar pjb3005 avatar roosterdragon avatar ygg01 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

linguini's Issues

Tracing bundle

Motivation

Ability to see what items aren't used in a resource. Things like SS14 would love to know if a translation is unused or missing.

What this means

Maintain some sort of data structure of elements that exist vs elements that don't. Design to be decided.

Improve docs

This is more of a meta issue, so contributors can expand docs and add it.

What docs are needed?

  • API Docs. Easiest to add needed on most public methods
  • How-To. Just a quick cookbook examples how to setup some Linguini in a specific case. No external dependencies needed.
  • Tutorial??. Possibly needed, but Linguini is relatively simple.
  • Onboarding docs for new contributors?? Not sure what people need.

Consider making errors nullable

Currently trying to resolve a message will always allocate a List<FluentError>, which is unnecessary in the majority of instances.

New line parsing not working correctly in 0.6.0

In 0.5.0, following CLRF formatted string:

settings-restart-ask-message = 
    Playnite needs to be restarted in order to apply new settings. Restart now?
    
    Restarting will cancel any active tasks (downloads) currently in progress.

was returned properly as

Playnite needs to be restarted in order to apply new settings. Restart now?

Restarting will cancel any active tasks (downloads) currently in progress.

but 0.6.0 now returns:

Playnite needs to be restarted in order to apply new settings. Restart now?
Restarting will cancel any active tasks (downloads) currently in progress.

Full file I'm adding as a resource via File.ReadAllText : en_US.zip

HasMessages does not work with attributes

You can only check by removing attributes first:

var attributes = key.LastIndexOf('.');
if (attributes > 0)
	return bundle.HasMessage(key.Remove(attributes));
else
	return bundle.HasMessage(key);

Setup Mono Pipeline

Motivation

Don't break Mono environments, plus let me sleep at night a bit easier.

Just run this under Linux in Mono. Simple but details are tricky.

Too many bundles

Good afternoon. There are about fifty enumerations in my ASP.NET Core Web API project. For each such enumeration, I have allocated ten language ftl files. Plus separate ftl contexts for general translations, separate ftl's for Exceptions. Everything is neatly arranged in folders. This ensures ease of translation management.

How to use Linguini correctly with this project structure?

Separate Serialization into its own package

Motivation

Right now, serialization from System.Text.Json is only used for tests. People probably don't need it in everyday usage. It also causes issues for Mono users.

Note

This change should bump minor, version. Due to API incompatiblity.

Reported line numbers are wrong

While working on SS14, I noticed that lines reported in ErrorSpan are wrong.

bible-test = tes
# comment
bible-heal-fail-others = {CAPITALIZE(THE$user))} hits {THE($target)} with {THE($bible)}, and it lands with a sad thack, dazing them!

would report error in bible-heal-fail-others = {CAPITALIZE(THE$user))} as error on line 1.

Add caching to Localization

Motivation

To further speed-up access, we need to allow the memoization/caching of values.

What this means

Right now when you access a Bundle, the system will always run a message finding algorithm. We can do better.

For a given Locale, message-id, and variables (aka Input) we can check if the value isn't already in the Dictionary<Input, String> if it is, we return it otherwise we run the resource finding algorithm.

error CS8032 when referencing the library under Mono

Run mono --version
Mono JIT compiler version 6.12.0.122 (tarball Mon Feb 22 17:33:28 UTC 2021)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	Interpreter:   yes
	LLVM:          yes(610)
	Suspend:       hybrid
	GC:            sgen (concurrent by default)

Compiling in Debug mode...
  Determining projects to restore...
  Restored /home/runner/work/OpenRA/OpenRA/OpenRA.Mods.Common/OpenRA.Mods.Common.csproj (in 3.[5](https://github.com/OpenRA/OpenRA/runs/6147316138?check_suite_focus=true#step:3:5)7 sec).
  Restored /home/runner/work/OpenRA/OpenRA/OpenRA.Mods.Cnc/OpenRA.Mods.Cnc.csproj (in 47 ms).
  Restored /home/runner/work/OpenRA/OpenRA/OpenRA.Launcher/OpenRA.Launcher.csproj (in 42 ms).
  Restored /home/runner/work/OpenRA/OpenRA/OpenRA.Game/OpenRA.Game.csproj (in 39 ms).
  Restored /home/runner/work/OpenRA/OpenRA/OpenRA.Utility/OpenRA.Utility.csproj (in 38 ms).
  Restored /home/runner/work/OpenRA/OpenRA/OpenRA.Server/OpenRA.Server.csproj (in 38 ms).
  Restored /home/runner/work/OpenRA/OpenRA/OpenRA.Platforms.Default/OpenRA.Platforms.Default.csproj (in 1.1 sec).
  Restored /home/runner/work/OpenRA/OpenRA/OpenRA.Mods.D2k/OpenRA.Mods.D2k.csproj (in 47 ms).
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/[6](https://github.com/OpenRA/OpenRA/runs/6147316138?check_suite_focus=true#step:3:6).0.3/analyzers/dotnet/roslyn3.11/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Game/OpenRA.Game.csproj]
CSC : error CS[8](https://github.com/OpenRA/OpenRA/runs/6147316138?check_suite_focus=true#step:3:8)032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn4.0/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Game/OpenRA.Game.csproj]
  OpenRA.Game -> /home/runner/work/OpenRA/OpenRA/bin/OpenRA.Game.dll
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn3.11/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Mods.Common/OpenRA.Mods.Common.csproj]
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn4.0/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Mods.Common/OpenRA.Mods.Common.csproj]
  OpenRA.Mods.Common -> /home/runner/work/OpenRA/OpenRA/bin/OpenRA.Mods.Common.dll
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn3.11/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Mods.Cnc/OpenRA.Mods.Cnc.csproj]
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn4.0/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Mods.Cnc/OpenRA.Mods.Cnc.csproj]
  OpenRA.Mods.Cnc -> /home/runner/work/OpenRA/OpenRA/bin/OpenRA.Mods.Cnc.dll
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn3.11/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Utility/OpenRA.Utility.csproj]
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn4.0/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Utility/OpenRA.Utility.csproj]
  OpenRA.Utility -> /home/runner/work/OpenRA/OpenRA/bin/OpenRA.Utility.dll
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn3.11/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Server/OpenRA.Server.csproj]
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn4.0/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Server/OpenRA.Server.csproj]
  OpenRA.Server -> /home/runner/work/OpenRA/OpenRA/bin/OpenRA.Server.dll
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn3.11/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Mods.D2k/OpenRA.Mods.D2k.csproj]
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn4.0/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Mods.D2k/OpenRA.Mods.D2k.csproj]
  OpenRA.Mods.D2k -> /home/runner/work/OpenRA/OpenRA/bin/OpenRA.Mods.D2k.dll
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn3.11/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Platforms.Default/OpenRA.Platforms.Default.csproj]
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn4.0/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Platforms.Default/OpenRA.Platforms.Default.csproj]
  OpenRA.Platforms.Default -> /home/runner/work/OpenRA/OpenRA/bin/OpenRA.Platforms.Default.dll
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn3.11/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Launcher/OpenRA.Launcher.csproj]
CSC : error CS8032: An instance of analyzer System.Text.Json.SourceGeneration.JsonSourceGenerator cannot be created from /home/runner/.nuget/packages/system.text.json/6.0.3/analyzers/dotnet/roslyn4.0/cs/System.Text.Json.SourceGeneration.dll : Could not resolve the signature of a virtual method. [/home/runner/work/OpenRA/OpenRA/OpenRA.Launcher/OpenRA.Launcher.csproj]
  OpenRA.Launcher -> /home/runner/work/OpenRA/OpenRA/bin/OpenRA.dll
make: *** [Makefile:[10](https://github.com/OpenRA/OpenRA/runs/6147316138?check_suite_focus=true#step:3:10)1: check] Error 1
Error: Process completed with exit code 2.

Attributes are not parsed if they begin with a tab character

Ffl-file with tabs for attributes:

2024-01-19_224956

Result of parsing (Linguini incorrectly identifies this as Junk):

2024-01-19_224910

Ftl without tabs (no errors, attributes were defined correctly):

2024-01-19_225124

Result of parsing:

2024-01-19_225346

Not sure if this will be correct, but my suggestion is to adjust the condition on line 400:
if (_reader.TryPeekChar(out var c) && c.IsAsciiAlphabetic() || c == '\u0009')

2024-01-19_232659

Errors differ from Fluent playground

For example following entry

bible-heal-fail-others = {CAPITALIZE(THE$user))}

Would fail on playground:

{
    "type": "Resource",
    "body": [
        {
            "type": "Junk",
            "annotations": [
                {
                    "type": "Annotation",
                    "code": "E0003",
                    "arguments": [
                        ")"
                    ],
                    "message": "Expected token: \")\"",
                    "span": {
                        "type": "Span",
                        "start": 40,
                        "end": 40
                    }
                }
            ],
            "content": "bible-heal-fail-others = {CAPITALIZE(THE$user))}",
            "span": {
                "type": "Span",
                "start": 0,
                "end": 48
            }
        }
    ],
    "span": {
        "type": "Span",
        "start": 0,
        "end": 48
    }
}

while on Linguini it would return with a different error.

Fix `NUMBERS` and `DATETIME` Builtins

Motivation

We don't have a good NUMBERS and DATETIME builtins function. It's modelled after fluent.rs, but there it's more of a stub, until icu is adopted.

What this means

This is a hard problem, there aren't any solution that can fit 1-to-1 to those built-ins. So it will need some plumbing, relying on C# or writing it from scratch.

Add Message Reference (Linguini extension of Fluent)

Motivated by space-wizards/RobustToolbox#2273
Seeing discussion in:

Problem

Let's say we have following message:

-cat = {$number ->
  *[one] Cat
  [other] Cats
}
-dog = {$number -> 
  *[one] Dog
  [other] Dogs
}

# We want message to be X attacked Y
attack-log-cat1-cat1 = Cat attacked cat.
attack-log-cat1-cat2 = Cat attacked cats.
attack-log-dog1-cat1 = Dog attacked cat.
attack-log-dog1-cat2 = Dog attacked cats.
# Etc.

This leads to combinatorial explosion. Instead we want something like

attack-log = { $$attacker(number: $atk_num) } attacked {$$defender(number: $def_num)}.

And then we pass following values into bundle

{
  "attacker": "-dog",
  "atk_num": 1,
  "defender": "-dog",
  "def_num": 2
}

Solution

I've decided to move the needle by:
A) Introducing a new fluent type called LinguiniReference
B) Introducing experimental syntax for start $$attacker which must be a Linguini reference. Plus allowing complex nesting like
$$attacker(number: $$something.Num) and access to attributes like so $$msg_id.Value
C) Create fast serialization of messages

Note: This is experimental syntax and will be probably changed depending on how it works out.

Refactor `FluentBundle` into several classes with `IFluentBundle`

Idea: to avoid ContainsKeys() and enable extra bundle types. Using TryAdd on IDictionary is not considered safe.

Implementation:

  1. Extract common FluentBundle operations to IFluentBundle interface.
  2. Add optional net8.0 support.
  3. Add three implementations - ConcurrentBundle, NonConcurrentBundle and FrozenBundle.

NOTE: This is a breaking change.

Any possibility of .NET 4.6.2 support?

Hi! I'm interested in using Fluent in a plugin for Playnite. Playnite currently requires .NET 4.6, and although there are plans to update to .NET 6+ in the future, there's no concrete timeline yet.

Would it be feasible to support .NET 4.6.2 in Linguini? I tried setting <TargetFrameworks>net4.6.2;netstandard2.1</TargetFrameworks> in Linguini.Shared to see what would happen, and it looks like it's relying on a few newer features:

  • NotNullWhen
  • NotNullWhenAttribute
  • ReadOnlySpan
  • ReadOnlyMemory

There's also this:

C:\tmp\Linguini\Linguini.Shared\Types\Bundle\IFluentType.cs(18,14): error CS8701: Target runtime doesn't support default interface implementation. [C:\tmp\Linguini\Linguini.Shared\Linguini.Shared.csproj]
C:\tmp\Linguini\Linguini.Shared\Types\Bundle\IFluentType.cs(32,14): error CS8701: Target runtime doesn't support default interface implementation. [C:\tmp\Linguini\Linguini.Shared\Linguini.Shared.csproj]

If this is something you're open to considering, I'd be happy to try to help (although .NET isn't my specialty).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.