openacg-group/cocoa

A general-purposed 2D rendering framework & visual novel engine.

cocoa's Issues

Documentations tracking

Trace improvements and additions of documentations

[typescript] Implement RasterPipeline, an image processing API

Plan to implement RasterPipeline, which is an image processing API based on native Skia API, and optionally integrate OpenCV WASM. This API allows users to generate and process images by creating a DAG (Directed Acyclic Graph) pipeline. Each node in the DAG could process or generate images, and they are connected together so that a node can take another node's output as its input. This is very similar to AVFilterDAG class in utau module, which process video or audio frames by a DAG constructed by ffmpeg's DSL (AVFilterDAG.MakeFromDSL()). But RasterPipeline is for static images, which means it is simplified and does not consider some complex things in multimedia processing, like frame timing. And RasterPipeline is designed for glamor module, supporting Skia APIs instead of VideoFrame or AudioFrame API in utau module.

Development of RasterPipeline is in progress.

Use rasterization cache to improve rendering performance

Cocoa has planned to use RasterCache machanism in the painting stage of layer tree to speed up rendering.

Integrate CanvasKit and paint with it

Track the process of CanvasKit integration.

• Fix the bug that glyphs cache causes serious memory pressure when deserializing an SkPicture containing fonts from CanvasKit.
• Fix the bug that the memory usage still increases slowly for an unknown reason.
• Improve the performance of large image deserialization (although uploading the image to TextureManager and rendering it by TextureLayer can avoid this problem, we still need to improve the performance of direct image drawing as much as possible).

CanvasKit is deprecated and Skia native APIs are available

WebAssembly-based JavaScript module CanvasKit is removed from the project. Instead, Skia native APIs are now available for less overhead while painting.

CanvasKit has been the painting interface for JavaScript in Cocoa for a long time. However, for many reasons, we have to remove it from the repository. Users can still compile a canvaskit.wasm by themselves and import it as a common WebAssembly module to use it. Here are the most important reasons why we deprecate it:

• Integration Glamor has many native classes exported from native Skia library, such as CkImage, CkPicture, CkBitmap and so on, while CanvasKit also has its corresponding classes SkImage, SkPicture, and SkBitmap. Users must do many things to convert between those data types. For example, when you try to write a simple music player by Cocoa, you need to use Utau module to extract and decode the cover image from mp3 file. Utau decoder gives you a VideoBuffer object, which can be converted to Glamor's CkImage object. Supposing that you want to preprocess it using CanvasKit (e.g. apply a Gaussian blur filter), then you must firstly convert the CkImage object to CanvasKit's SkImage object before preprocessing it. The preprocessed image should be rendered on the window, which means you need to convert the preprocessed image back to Glamor's CkImage object. That is quite painful and complex.
• Performance As WASM modules need to be compiled at runtime, applications that use CanvasKit module spend more time starting up. What's worse, as the example shown above, converting between Glamor data types and CanvasKit data types causes pixels copying. Meanwhile, SkPicture objects generated by SkPictureRecorder also must be converted to Glamor's CkPicture before adding it to the Glamor's layer tree. Converting between them is implemented by serializing and deserializing the picture object. If the picture object contains typefaces or large images, the conversion is an expensive operation. We have implmented a complicated mechanism to reduce the overhead for typefaces (#3), but for large images, it is a quite difficult problem.
• Cache Skia caches many rendering objects like shaders, blenders, image filters, etc., but creating CkPicture from deserializing breaks the cache mechanism.

Now with the native Skia APIs, object conversions are not needed anymore, and pixels copying can be reduced to the minimal value. For the music player example shown above, pixels will be copyed N time(s) before it is submitted to Blender or landed on a certain surface:

• N=0 if the decoded frame is already in BGRA format;
• N=1 if the decoded frame is in YUV format or other formats.