achirkin/easytensor

Many-dimensional type-safe numeric ops

Easytensor: many-dimensional type-safe numeric ops

The project consists of two parts:

• dimensions is a library to support type-level operations on lists of dimensions;
• easytensor wraps low-level operations on primitive byte arrays in a type-safe data type indexed over an element type and a list of dimensions.

dimensions

Data.Type.List and Data.Type.Lits provide type-level operations on lists, Nats, and Symbolss.

Numeric.TypedList is the core module providing a typelist-indexed type that is, in fact, just a newtype wrapper on a plain haskell list. The TypedList (f :: k -> Type) (xs :: [k]) represents a lot of things; by changing type parameter f, I use TypedList as a flexible (yet typesafe) tuple, a finite dimensions list, or an index over such a list.

Numeric.Dimensions provides:

• promoted type XNat = N Nat | XN similar to type Maybe; kind XNat is used for type variables when some of dimensions in a type-level list are not known at compile time;
• data type Dims (ds :: [k]), where k is either Nat or XNat, together with class Dimensions ds it allows lots of type-level operations on type-level dimensionality;
• data type Idxs (ds :: [k]) is used to index over many-dimensional space defined by Dims ds.

easytensor

This library aims at providing fast, simple, and useful geometry types for use in computer graphics and low-dimensional physics modelling.

All geometry types implement standard Prelude numerical classes, such as Num, Fractional, Floating, lexicographical Ord favouring element-wise operations: functions like +, *, / are all element-wise. Common matrix and vector operations are provided separately.

Data type DataFrame is presented in two versions:

• DataFrame t (ds :: [Nat]) - dimensionality of a dataframe is totally known at compile time.
• DataFrame t (ds :: [XNat] - some dimensions may be known at runtime only.

Parameter t of a DataFrame can be arbitrary type that has an instance of PrimBytes. PrimBytes typeclass can be automatically derived using Generics. This mechanics allows creating interleaved arrays, e.g. DataFrame (Double, Float, Float) ds.

Parameter t of a DataFrame can also be a list of PrimBytes types: this way, DataFrame consists of several "columns" or "variables" of different types and same dimensionality.

Behind the scenes all data types are implemented as primitive values or primitive byte arrays, aiming at maximum performance. Tricky layers of newtypes and closed type families (which are not exposed to a user) allow some kind of ad-hoc polymorphism: for example, Vector t n implemented as ByteArray# is overloaded by a specialized FloatX2 Float# Float# in case of Vector Float 2.

Supported GHC versions

The packages are tested on GHC 8.4+. dimensions may work on GHC 8.2, but the corresponding tests were dropped. Support of easytensor on GHC 8.2 was dropped due to:

• https://gitlab.haskell.org/ghc/ghc/issues/14058
• https://gitlab.haskell.org/ghc/ghc/issues/13188
• Annoying Semigroup-Monoid story

What have changed since version 1

Everything! In general, v2 of easytensor is meant to show an expectable behavior:

• All Show and Read instances look like automatically-generated instances for algebraic data types.
• 0-based indexing instead of 1-based indexing makes conversion between offsets, indices, and Enum more intuitive.
• Order of dimensions in Dims list is reversed (the first dimension is "the most significant"), thus making Ord instances of Dims and Idxs coincide with conventional Haskell lexicographic ordering. The implication of this is the next two points.
• DataFrame Ord instances are now proper total lexicographic ordering.
• DataFrame layout now is row-first instead of column-first. Therefore, to keep low-level SIMD optimizations of 3D geometry possible, I've had to transpose all matrices in HomTransform4 class.
• Nat-indexed Dims, Idxs, and DataFrame now have Generic and Data instances that make these datatypes look like ordinary algebraic data types.
• More obvious ways to construct DataFrames from pieces.
• Removed all declarations and modules that may look controversial or do not belong here.
• Just added more tests and focused on the core functionality and correctness :)

easytensor's Issues

stack build error

Building with OSX, easytensor-0.4.0.0 and nightly-2017-11-07 gives me:

src-base/Numeric/Array/Family/ArrayD.hs:56:37: error:
         error: '##' cannot appear at end of macro expansion
       |
    56 | #define EL_ZERO                  0.0##
       |                                     ^
    #define EL_ZERO                  0.0##

# gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

easytensor-vulkan doesn't build

I just got back into vulkan coding after a 6-month hiatus, so I haven't been following all the changes you've been making... I upgraded to the latest easytensor and vulkan-api, and when I built them, easytensor-vulkan failed with this error:

src/Graphics/Vulkan/Marshal/Create/DataFrame.hs:184:10: error:
    Not in scope: type constructor or class ‘VulkanMarshalPrim’
    Perhaps you meant ‘VulkanMarshal’ (imported from Graphics.Vulkan.Marshal.Internal)
    |
184 | instance VulkanMarshalPrim (VkDataFrame t ds) where
    |          ^^^^^^^^^^^^^^^^^

dimensions-0.3.0.0 fails its test suite

Citing from https://nix-cache.s3.amazonaws.com/log/bva35jjj8g57dp80cxd0ds11x76msq16-dimensions-0.3.0.0.drv:

Running 1 test suites...
Test suite dimensions-test: RUNNING...
Prelude.read: no parse

Need PrimBytes instance for `FunPtr a`

There's currently no such instance. I'll work on a patch..

packDF usage?

The following works:

myVector :: DataFrame Double '[4]
myVector =
  packDF @Double @4 @'[] 1 2 3 4

The following doesn't type-check

vertices :: DataFrame Vertex '[4]
vertices =
  packDF @Vertex @4 @'[]
    (Vertex (vec3 0 0 0) (vec2 0 0))
    (Vertex (vec3 0 0 0) (vec2 0 0))
    (Vertex (vec3 0 0 0) (vec2 0 0))
    (Vertex (vec3 0 0 0) (vec2 0 0))

data Vertex =
  Vertex {
    vtxPos :: Vec3f,
    vtxTexCoord :: Vec2f
  } deriving (Eq, Show, Generic)

instance PrimBytes Vertex

The error is:

    • Couldn't match type ‘DataFrame Vertex '[]’ with ‘Vertex’
      Expected type: Vertex
                     -> Vertex -> Vertex -> Vertex -> DataFrame Vertex '[4]
        Actual type: PackDF Vertex '[] 4 (DataFrame Vertex '[4])
    • The expression ‘packDF @Vertex @4 @'[]’
      is applied to four arguments,
      but its type ‘PackDF Vertex '[] 4 (DataFrame Vertex '[4])’ has none
      In the expression:
        packDF
          @Vertex
          @4
          @'[]
          (Vertex (vec3 0 0 0) (vec2 0 0))
          (Vertex (vec3 0 0 0) (vec2 0 0))
          (Vertex (vec3 0 0 0) (vec2 0 0))
          (Vertex (vec3 0 0 0) (vec2 0 0))
      In an equation for ‘vertices’:
          vertices
            = packDF
                @Vertex
                @4
                @'[]
                (Vertex (vec3 0 0 0) (vec2 0 0))
                (Vertex (vec3 0 0 0) (vec2 0 0))
                (Vertex (vec3 0 0 0) (vec2 0 0))
                (Vertex (vec3 0 0 0) (vec2 0 0))
    |
295 |       packDF @Vertex @4 @'[]

Why?

export convenience contraints for >1 KnownDim and Dimensions

dimensions-1.0 is excellent and has removed all of the helpers I've written for hasktorch. The remnants of these are just KnownDim<N> and Dimensions<N> convenience constraints since we are prioritizing statically typed tensors, and we wind up with a high number of these one-off constraints (I've only gotten up to 2d auto differentiable convolution, and I've found KnownDim5 fairly useful). Would this be something that you might consider adding to dimensions?

instances for ad package missing

Hi I would like to use easytensor with ad but instance Quaternion (Reverse s Double) is missing. Perhaps the methods of class Quaternion can be ordinary functions on newtype Quater a = Quater (Vector a 4) with SPECIALIZE pragmas for Double and Float. Then Numeric.Quaternion.Internal.QDouble and QFloat duplication will be gone, and maybe the numbers defined by ad will also work.

slice and ewfoldr usage

I'm trying to understand how to use the DataFrame operations, and need some help.

What I specifically need for my code right now is, given a Vector VkMemoryHeap 16, I need to get the first N elements of that vector (determined at runtime). Then I need to filter them using a predicate. Then of the remaining ones, I need to sum their getField @"size"values.

I used to do this by converting the Vector to a list, and then just use standard list operations, but the toList function is no longer exported by easytensor.

I was looking at slice, because it seems relevant, and I can't make sense of what it's trying to do, or how to use it.

ghc-8.6

prepping for ghc-8.6, following https://gist.github.com/DanBurton/43b5f5155fdd1affd02f6e86f37da9ae, I get:

/Users/tonyday/haskell/easytensor/dimensions/src/Numeric/Dim.hs:65:44: error: parse error on input ‘*’
       |
    65 |   , Nat, CmpNat, type (+), type (-), type (*), type (^)
       |                                            ^

Some sort of interaction between * and Type perhaps?

HomTransform4 not implemented

I'm at the point in the vulkan tutorials where I have need of these transforms. I'd like to make an attempt at implementing them, but I'm unclear on the intent behind and plan for the existing HomTransform4 class. Basically, why is it a class at all, rather than just having free functions that are constrained to Floating t => or something? What other instances would there be besides Float and Double?

`dimensions` package is missing a CHANGELOG

dimensions package is missing a CHANGELOG

PrimBytes instances for Foreign.C.Types

For PrimBytes to be applied to FFI, instances for these C-types would be nice to have.

Tutorial / walkthrough please

These libraries (dimensions / easytensor) look useful and interesting. Unfortunately it's not clear how to get started. Some examples / tutorial would be much appreciated!

Why is Idx bounded to 0 < i =< Dim d?

Wouldn't 0 =< i < Dim d be more programmer-friendly? I realize there is an escape hatch with fromEnum, but I'm wondering if there is a specific reason behind this interface.

Slicing a DataFrame

Is there some way to take, say, a Vector _ 10 and drop the last 5 elements to get a Vector _ 5? In Vulkan, there are cases where the size of the array that Vulkan returned is much larger than the number of actual valid elements.

For example, within VkPhysicalDeviceMemoryProperties, the memoryHeaps field is an array that always has length VK_MAX_MEMORY_HEAPS, but only the first memoryHeapCount elements are actually valid.

I would like to be able to, say, fold over these elements. As it stands, the only way I can see to do it is to first convert it to a list, and then use the standard list functions for that.

cabal files missing

I don't see cabal files for easytensor anymore, so I can't build it. I'm guessing they're optional when using stack? Problem is that I don't use stack.

Need a better error message for absence PrimBytes (DataFrame t ds) instance

When a code requires PrimBytes (DataFrame t ds), GHC uses a
deriving instance PrimBytes (Array t ds) => PrimBytes (DataFrame t ds),
which, in turn, requires PrimBytes (Array t ds).
Array t ds is a type family; only individual members of the type family have the instance.
To make things worse, there are other instances of PrimBytes defined in easytensor-vulkan.
And GHC starts to panic (Overlapping instances for PrimBytes).

Need to try to reconsider a combination of TypeError and OverlappingInstances to improve this.

The issue comes from a comment #4 (comment).

Stability of LU factorization

Sometimes LU factorization is very unstable, especially for Float matrices.
We should fix this some day.

easytensor/easytensor/test/Numeric/MatrixFloatTest.hs

Lines 80 to 90 in 4cc9a38

	-- TODO: improve lu!
	-- lu factorization is very unstable in some cases
	--
	-- >>> m = unsafeFromFlatList (Dims @'[4,4]) 0 [1,0,1,-2, 0,-1,1,2, 2,-4,2,-2, 4,-8,-2,2 ] :: Mat44f
	-- >>> det m == 0
	-- >>> det (transpose m) == 60
	prop_LU :: SomeSquareMatrix NonSingular TestElem -> Bool
	prop_LU (SSM m)
	= let f = lu m
	aeq a b = maxElem (b - a) <= eps * maxRows m
	in aeq (luPerm f %* m) (luLower f %* luUpper f)

prop_Comparisons fails every once in a while

A QuickCheck test prop_Comparisons fails from time to time. QuickCheck generates DataFrames randomly, 100 tests for each property by default. Usually, this kind of problem happens due to floating point precision errors. But in this case, it is hard to find out what can cause such error.

`dimensions-1.0.1.1` fails to build with GHC 8.10 and up

dimensions-1.0.1.1$ cabal build -w ghc-8.10.7
...
src/Numeric/Dim.hs:123:29: error: Not in scope: type variable ‘k’
    |
123 | pattern Dim :: forall (n :: k) . () => KnownDim n => Dim n
    |                             ^

Hello, Artem Chirkin. I have some question for easytensor.

Hi, I am developing Vulkan in Haskell using easytensor and vulkan-api library.

Thank you for your contribution.

If vulkan-api and easytensor do not exist, Haskell's vulkan programming is not possible.

But easytensor is so hard for me to understanding... and English too.. T_T

I have a question.

I want to get pure List from XFrame as below but I had failed..

dataFrameToList :: DataFrame Vec3f (xns::[XNat]) -> [Vec3f]

Can you help me? Let me know how can I convert XFrame to List.

And I want to know how to apply map function to XFrame.

I will wait for your answer. Thank you.

And I hope more easy examples about easytensor.

Below are some of my project and easytensor notes.

https://github.com/ubuntunux/HulkanEngine3D
https://github.com/ubuntunux/PyEngine3D
https://wikidocs.net/47790

Projection matrices need variants for Vulkan and have wrong docs

In Numeric.Matrix: perspective and orthogonal currently follow OpenGL conventions. They require negative near and far values, otherwise they produce bogus Z values and messed up signs. The haddock comment on perspective states "Near plane clipping distance (always positive)" which is wrong. I suppose the OpenGL convention sees them as z-coordinates of the planes in screen space instead of distances to the camera.

They also project onto Z values from -1 to 1. For Vulkan, they need to produce a Z range from 0 to 1.

The GLM library has different defines for those variants, but I don't believe compile time flags are a good solution. I suggest splitting the projections up into variants and naming them accordingly.

Missing pattern synonyms in haddock output

Not sure if that's a limitation of haddock, but pattern synonyms like Vec2 don't appear in the haddock output of Numeric.Vector, despite being re-exported there. They should be mentioned somewhere, as they are needed for comfortable deconstruction :)

internal error: ARR_WORDS object entered

Function getBytes on ArrayBase type can crash with the following message:

internal error: ARR_WORDS object (0x...) entered!
    (GHC version 9.0.1 for x86_64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug

I have a workaround for that, but sadly it does not work anymore with GHC 9.0.1.

Here is the related issue:
https://gitlab.haskell.org/ghc/ghc/-/issues/19645

Add mechanism for easily getting field offsets

I'm not really sure how this would work, exactly, but it seems like there should be some mechanism for easily getting the offset of a field in a PrimBytes instance. In my code I have this:

data Vertex =
  Vertex {
    vtxPos :: Vec3f,
    vtxColor :: Vec3f,
    vtxTexCoord :: Vec2f
  } deriving (Eq, Show, Generic)

instance PrimBytes Vertex

vertexAttributeDescriptionsAt :: Word32 -> [VkVertexInputAttributeDescription]
vertexAttributeDescriptionsAt binding =
  createVk @VkVertexInputAttributeDescription <$> [
    set @"binding" binding &*
    set @"location" 0 &*
    set @"format" VK_FORMAT_R32G32B32_SFLOAT &*
    set @"offset" 0,

    set @"binding" binding &*
    set @"location" 1 &*
    set @"format" VK_FORMAT_R32G32B32_SFLOAT &*
    set @"offset" offset1,

    set @"binding" binding &*
    set @"location" 2 &*
    set @"format" VK_FORMAT_R32G32_SFLOAT &*
    set @"offset" offset2
  ]
  where
    offset1 = fromIntegral (sizeOf $ vtxPos undefined)
    offset2 = offset1 + fromIntegral (sizeOf $ vtxColor undefined)

It'd be nice if I could just say something like offsetOf @"vtxColor". (And for that matter, why not sizeOf @"vtxColor" or just sizeOf @Vec3f or something.) Edit: Eh, not that sizeOf @Vec3f is, itself, bad, but you wouldn't want to do that to get the size of the vtxColor field, in case the type of that field changes.

Type-checking yet incorrect typo in Dim.hs

I think the right-hand side here should use sameDim, not sameDim': https://github.com/achirkin/easytensor/blob/master/dimensions/src/Numeric/Dim.hs#L321

Need simpler way to get a PrimBytes field from a Ptr

If I have a Ptr Foo, where Foo has a bar :: Word32 field, I can get that field's value like:

bPeekByteOff @Word32 fooPtr $ bFieldOffsetOf @"bar" (undefined :: Foo)

It would be nice to not have to explicitly specify the type of the field when getting it, as that should be determinable by the field name. It would also be nice to have a single function that covers this, like readField from Graphics.Vulkan.Marshal.