zpl-c/zpl

📐 Pushing the boundaries of simplicity

zpl's Issues

opts module depends on json module

The opts module depends on json due to the usage of zpl__skip and zpl__trim, both of which are macros defined in the json module.

Improve heuristics for the job system's job slot allocation

Allocate a group of job slots based on the average job load. Scale down job slots when average load is lower than a specific threshold (or be flexible about it)

The goal is to minimize number of allocations when batch of jobs arrive while being slot limited.

Multi-CU builds fail on inline methods under qmake build system.

I am getting a lot of errors related to inline methods when using Qt Creator's qmake build system for my test C application.

This is a multi-compilation unit project.

Re-write bitstream

Re-write bitstream interface.

User should work with a structure containing a pointer to the data.

Arch: Fix ARMv7 builds

zpl_thread join/is_running

in zpl_thread_join, pthread_thread_join should be called unless is_running.
zpl_thread.is_running will be true if 'zpl__thread_proc' returns before 'zpl_thread_start_with_stack' ends.

Research: ARM threading module support

Figure out how to implement the threading module for commonly supported ARM architectures.

Feature: Implement a TOML parser, conformant to the latest standards

I absolutely love zpl! I use it for every C based project, and it brings over many of the things I enjoy from C++, while also benefitting from the simplicity of C.

Would it be possible if a TOML parser could be embedded? You could take a look at tomlc99 for reference.

CI: Fix Discord avatar urls

text parser: deprecate old JSON module

Blocked by #73

Iterating through a hashtable

Hello,

There seems to be no function or standard way to do it (sorry if I missed something obvious). Looking at *_find's implementation, I tried to replicate it, but it caused segfaults (I didn't dived too much into it fearing I might be doing completely wrong thing here, and I am not familiar with the design of this particular hash table implementation).
So my suggestion: something like a void *_map(void (*f)(VALUE value)) function could work out great in such scenarios.

Thank you. Have a nice day!

provide a non-floating-based version of `zpl_time_now`

which may returns zpl_u64 (not zpl_f64).

Edit: for portable and efficient reasons.

text parser: integrate JSON/JSON5/SJSON support

zpl_printf doesn't support %0d correctly

ZPL leaves at least one 0 character in the output when a format specifier %0d is used.

Example:

zpl_printf("0.%0d%d\n", 0, 32);
// Expected output: 0.32
// Actual output:   0.032

Cross-platform directory support

This should be a major feature.
Ref: https://github.com/tronkko/dirent
Thanks.

Apply changes from gingerBill/gb

gingerBill/gb@7bba49c

Cross Compile error on ARM

Hello, I've tried cross-compiling the example I'd shown in issue #34.
Cross compile toolchain and Linux OS is one built from Buildroot, with GCC version 6.4.0 (Target: arm-bulidroot-linux-gnueabihf, Arch: armv7 (32bit)).

The error:

zpl.h:553:35: fatal error: emmintrin.h: No such file or directory

zpl.h:

550 #if !defined(ZPL_NO_SIMD)
551     #if !defined(ZPL_SYSTEM_ANDROID) && !defined(ZPL_SYSTEM_IOS)
552         #if !defined(ZPL_SYSTEM_EMSCRIPTEN)
553             #include <emmintrin.h>
554         #elif defined(ZPL_CPU_X86) && !defined(ZPL_SYSTEM_EMSCRIPTEN)
555             #include <xmmintrin.h>
556         #else
557             #include <sched.h>
558         #endif
559     #endif
560 #endif

sched.h exist on this platform, so I modified the header to have it take this path in the above snippet. This caused new compile errors about Unknown architecture. The main README mentions ARM is supported.

Add thread pool

This would be helpful for my project and others.
Thanks.

JSON Schema

Implement JSON Schema validator.

http://json-schema.org/example1.html

Unicode support

Would unicode support for zpl ever be considered? I'm sure it wouldn't be massively complicated, at least on the Windows side of things.

text parser: integrate TOML support

CI: Set up ARMv7 tests

tests: rewrite json and csv tests so that memory is always freed if case fails

text parser: implement base library

Regarding addition of extra matrix functions which are required for non opengl environments

How will the API change in order to add Left handed coordinate system based matrix manipulation functions and Perspective and orthographic functions which allow a NDC range of [0..1] (eg. Direct3d)?

Feature: Implement VFS

Based on #65, we could implement a virtual file system abstraction to allow for the file module to be backed by additional storage options. The primary goal is to ensure that all OS-dependent calls regarding file operations would be abstracted away so that there is more room for custom backing to be introduced. For instance, users could create/rename/delete files in a TAR archive by using the very same API they would use in the case of an OS-backed file system.

The design is still a work in progress and requires various considerations to be taken into an account.

zpl_json.h - Add support of exponential numbers.

A more efficient way of using goto statement

The mix of the goto statement and the branch statement is often not a good practice. Because a branch if ... then A else B usually implies that exactly one of the A or the B will be executed. The code in README.md violates this rule.

zpl/code/apps/examples/json_benchmark.c

Line 35 in 0a0593f

goto bad;

if (ok) {
  if (!ok2) goto err;
} else {
  err:
}

consider to rewrite it as:

if (! ok) goto err:
if (! ok2) goto err:

err:

Argument/Option parsing w/positional arguments and flags only works with flags last

The argument and option parsing only seems to work when the flags are placed last
./dug example.com 127.0.0.1 -d
works while
./dug -d example.com 127.0.0.1
does not.

Output from first case:

hostname: example.com
dns_server: 127.0.0.1
debug_mode true

Output from from second case:

ERROR: Extra value for option ""
USAGE: ./dug [hostname] [dns_server]
OPTIONS:
-n, --hostname: hostname to get DNS query for
-s, --dns_server: dns server to query
-h, --help: the HELP section
-d, --debug: enable debugging

Code:

zpl_opts opts = {{0}};

zpl_opts_init(&opts, zpl_heap(), argv[0]);

zpl_opts_add(&opts, "n", "hostname", "hostname to get DNS query for", ZPL_OPTS_STRING);
zpl_opts_add(&opts, "s", "dns_server", "dns server to query", ZPL_OPTS_STRING);
zpl_opts_add(&opts, "h", "help", "the HELP section", ZPL_OPTS_FLAG);
zpl_opts_add(&opts, "d", "debug", "enable debugging", ZPL_OPTS_FLAG);
zpl_opts_positional_add(&opts, "dns_server");
zpl_opts_positional_add(&opts, "hostname");



int32_t opts_compile = zpl_opts_compile(&opts, argc, argv);
char *hostname;
char *dns_server;

if (opts_compile && zpl_opts_positionals_filled(&opts)) {
    // print help menu
    int32_t help = zpl_opts_has_arg(&opts, "help");
    if (help) {
        zpl_opts_print_help(&opts);
        return 0;
    }
    // enable debug mode
    debug = zpl_opts_has_arg(&opts, "debug");
    hostname = zpl_opts_string(&opts, "hostname", NULL);
    dns_server = zpl_opts_string(&opts, "dns_server", NULL);

}
else {
    zpl_opts_print_errors(&opts);
    zpl_opts_print_help(&opts);
    return 0;
}

Maybe I'm doing something wrong or it just works like that, not sure

Thanks!

Comparisons against EOF fail when compiled with -funsigned-char

See https://www.linuxtopia.org/online_books/an_introduction_to_gcc/gccintro_71.html

pthread_join must be called for PTHREAD_CREATE_JOINABLE thread

at source/threading/thread.c line 76:
if (!t->is_running) return;
t->is_running is set to false at the end of thread proc.
For pthread, a pthread_join call cannot be ignored for PTHREAD_CREATE_JOINABLE pthread.

zpl_random_range_f64 appears to generate garbage

Seeing the following on windows 10 with 12.2.0

#define ZPL_IMPLEMENTATION
#include "zpl.h"

int main()
    zpl_random rand;
    zpl_random_init(&rand);
    for (int i=0; i < 100; i++)
    {
        zpl_printf("%f\n", zpl_random_range_f64(&rand, 0.0, 1.0));
    }
    return 0;
}

This program, which I'd expect to print random numbers between 0 and 1, randomly prints either 0.0, -0.0, 2.0, or -2.0.

Am I misunderstanding the API here or is this an implementation bug?

Unsafe functions on windows

If you use zpl on windows (at least with MSVC) you get 2 unsafe code warnings because of tmpfile and getenv function calls. Maybe fix the warning with something like

#ifdef _WIN32
tmpfile_s(...);
#else
tmpfile(...);
#endif

Lines:

• zpl.h (9012)
• zpl.h (10272)

Tests

Coverage + documentation of several systems in helper library, platform library and image library.

Move old tests to deprecated folder.

Examples: Fix various compilation warnings

These appear in our examples when built using -O3 flag.

WIN32_LEAN_AND_MEAN macro redefinition warning

Because zpl defines as #define WIN32_LEAN_AND_MEAN 1 instead of #define WIN32_LEAN_AND_MEAN. as I think is the usual way (see example link from msdn), I get a lot of warning C4005: 'WIN32_LEAN_AND_MEAN': macro redefinition because another library I use is also defining this "the normal way". By omitting the trailing 1 this could be fixed.

Example: https://docs.microsoft.com/de-de/previous-versions/technical-content/ms741665(v=vs.85)

Atomics aren't atomic

The zpl_atomic32, zpl_atomic64 and zpl_atomic_ptr are all declared as volatile, and are used unprotected by any guards. Atomics and volatile are not the same thing, and writes to volatile variables may still have the following problems:

• torn reads/writes
• on writes, work done to other variables on that thread may not be published, leading another thread to think it has acquired access to a critical region of code, but the data the code operates in does not have the first thread's changes. Clarification below

Quotes from the wikipedia page for [volatile variables](https://en.wikipedia.org/wiki/Volatile_(computer_programming):

• "in C and C++ it does not work in most threading scenarios, and that use is discouraged"
• "volatile variables are not threadsafe in the vast majority of current implementations"
• "the usage of volatile keyword as a portable synchronization mechanism is discouraged by many C/C++ groups"
• "neither the C specification (before C11) nor the C++ specification (before C++11) specifies a multi-threaded memory model, so volatile may not behave deterministically across OSes/compilers/CPUs"
• "According to the C++11 ISO Standard, the volatile keyword is only meant for use for hardware access; do not use it for inter-thread communication"

This is not helped by the fact that some people perpetuate the myth that they're the same (referring to the M.M comment)

I don't know if there is a portable way of using atomics pre-C11, but volatile definitely isn't it.

To clarify my comment above, imagine there are two threads. The variable a is atomic, and g is global (not atomic). Both variables start out at 0. Thread 1 runs the following code

g = 1;
a = 1;

Thread 2 runs this code.

if (a == 1) {
    assert(g == 1);
}

If a is truly atomic, then the assertion in thread 2 must NEVER fail. However if a is only volatile, then the code may fail. In fact, if (a == 1) may not even run properly if Thread 2 gets a torn read on a, which can happen if a isn't atomic, only volatile.

If a is being used to guard critical resources, this leads to subtle, hard-to-detect and hard-to-reproduce errors.

Adding FreeBSD support

Please consider applying the following patch to code/zpl.h in order to add support for FreeBSD 12. Might be necessary to polish it a little beforehand, as I am not a C expert.

--- zpl.h	2019-11-01 05:35:57.095504000 +0100
+++ zpl_new.h	2019-11-01 05:50:39.201436000 +0100
@@ -520,10 +520,16 @@
 #include <stdlib.h> // NOTE: malloc on linux
 #include <sys/mman.h>
 
-#if !defined(ZPL_SYSTEM_OSX)
+#if !defined(ZPL_SYSTEM_OSX) && !defined(ZPL_SYSTEM_FREEBSD)
     #include <sys/sendfile.h>
 #endif
 
+#if defined(ZPL_SYSTEM_FREEBSD)
+    #include <sys/types.h>
+    #include <sys/socket.h>
+    #include <sys/uio.h>
+#endif
+
 #include <dirent.h>
 #include <sys/stat.h>
 #include <sys/time.h>
@@ -767,6 +773,11 @@
     #endif
 #endif
 
+#if defined(ZPL_SYSTEM_FREEBSD)
+    #define ZPL_EXTERN extern
+    #define zpl_inline inline
+#endif
+
 #if !defined(zpl_no_inline)
     #if defined(_MSC_VER)
         #define zpl_no_inline __declspec(noinline)
@@ -1557,7 +1568,7 @@
     zpl_isize threads_per_core;
 } zpl_affinity;
 
-#elif defined(ZPL_SYSTEM_LINUX) || defined(ZPL_SYSTEM_EMSCRIPTEN)
+#elif defined(ZPL_SYSTEM_LINUX) || defined(ZPL_SYSTEM_FREEBSD) || defined(ZPL_SYSTEM_EMSCRIPTEN)
 
 typedef struct zpl_affinity {
     zpl_b32   is_accurate;
@@ -6967,7 +6978,7 @@
 
 #else
 
-#if defined(ZPL_SYSTEM_LINUX)
+#if defined(ZPL_SYSTEM_LINUX) || defined(ZPL_SYSTEM_FREEBSD)
 zpl_inline zpl_f64 zpl__unix_getime(void) {
     struct timespec t;
     zpl_f64 result;
@@ -8152,7 +8163,7 @@
     return a->threads_per_core;
 }
 
-#elif defined(ZPL_SYSTEM_LINUX)
+#elif defined(ZPL_SYSTEM_LINUX) || defined(ZPL_SYSTEM_FREEBSD)
 // IMPORTANT TODO: This zpl_affinity stuff for linux needs be improved a lot!
 // NOTE(zangent): I have to read /proc/cpuinfo to get the number of threads per core.
 
@@ -10816,7 +10827,11 @@
     struct stat stat_existing;
     fstat(existing_fd, &stat_existing);
 
+#if defined(ZPL_SYSTEM_FREEBSD)
+    size = sendfile(new_fd, existing_fd, 0, stat_existing.st_size, NULL, 0, 0);
+#else
     size = sendfile(new_fd, existing_fd, 0, stat_existing.st_size);
+#endif
 
     close(new_fd);
     close(existing_fd);

%lld issue in JSON module

Replace these with PRid64 from inttypes.h. %lld is only valid for GNU C libraries.

Verify on mingw, msvc doesn't care about this issue however.

Time: questionable unconditional access of `PMCCNTR` under `__ARM_ARCH_7A__`

Hello!

I'm questioning the implementation of zpl_rdtsc() under #elif defined(__ARM_ARCH_7A__) in time.c.

Initial observations on a test program (compiled with no special flags) exercising a single call of that function (outputting its return value) on a Raspberry Pi 4 running Raspberry Pi OS (32-bit):

$ gcc -O0 -ggdb zpl_rdtsc_test.c -o ts_gcc
$ clang -O0 -ggdb zpl_rdtsc_test.c -o ts_clang

• GCC: compiles when the commit of PR #44 is applied, outputs 0
• Clang (as obtained from https://releases.llvm.org/download.html#10.0.1): compiles, aborts with "Illegal instruction"

The first thing to note is that GCC and Clang compile different #if blocks: #elif (__ARM_ARCH >= 6) for GCC and #elif defined(__ARM_ARCH_7A__) for Clang.
This is easily explained by looking at the macros that each of them pre-defines:

$ clang  -dM -E - </dev/null | grep ARM

has

#define __ARM_ARCH 7
#define __ARM_ARCH_7A__ 1

while the same invocation with gcc shows that by default, GCC targets ARM architectures upwards of version 6 instead.

The more interesting issue is the SIGILL. My question: what is the background for accessing PMCCNTR unconditionally under __ARM_ARCH_7A__?

According to the "ARM Architecture Reference Manual Armv7-A and Armv7-R edition", section "C12.9 Performance Monitor registers" (page 2314) -- more specifically, table C12-11 on page 2319 -- accessing the Cycle Count Register is still only allowed if the PMUSERENR.EN bit is set to 1, which in turn can not be set from PL0 (unprivileged) modes.

And, as the initial output of 0 suggests and an additional check confirms, PMUSERENR.EN as visible from userspace is indeed set to 0 on my system, at least without additional measures taken.

Presumably, the code in question does have a background [1], but then the preprocessor condition needs to be refined IMO.

[1] such a specific variant of the ARM architecture allowing implementations to omit the access control?

Look into implementing fiber API for zpl

Consider looking at https://code.byuu.org/libco as a reference source

zpl_random_range_i64 generates value out of range

zpl_i64 zpl_random_range_i64(zpl_random *r, zpl_i64 lower_inc, zpl_i64 higher_inc) {
zpl_u64 u = zpl_random_gen_u64(r);
zpl_i64 i = *cast(zpl_i64 *)&u;
zpl_i64 diff = higher_inc-lower_inc+1;
i %= diff;
i += lower_inc;
return i;
}

if lower_inc is 0 and higher_inc is 1, the function generates '-1' as return value.

Integrate zpl_json.h into ZPL core

Feature: Add archiving functionality (zip/tar/..)

Hey, just looked at the combined-header file and the available functionalities/modules at readme.

I thought maybe some kind of archiving functionality could fit well for this library as well, due there are File related functionalites. So quickly zip up a folder, or i.e. two files to create a backup of i.e. application configuration + some images.

Don't really implying any preference, whatever is easy and short (LOC/singleheader) to implement ... zip?tar?
Maybe minizip has something to offer.

Feel free to close if you don't think it fits for the project.

Add option to not clobber names that don't begin with `zpl_` or `ZPL_`

Currently #including this library drops a bunch of stuff that you might otherwise be using into the global namespace, like defining u8, i32, intptr, and most importantly a macro called cast. It also does #define NULL nullptr if compiled for C++, which may break backwards compatability with existing C code when used inside C++.

Can these be defined conditionally?

Memory leak in zpl_opts_free

zpl_opts_free does not free zpl_opts_entry->text (if zpl_opts_entry->type is ZPL_OPTS_STRING).

zpl_opts_entry *e = 0;
for (int i = 0; i < zpl_array_count(opts->entries); ++i) {
    e = opts->entries + i;
    if (e->type == ZPL_OPTS_STRING) { 
        zpl_string_free(e->text);
    }
}

Compile issues

I'm trying to compile a simple example on Linux with gcc 7.4.0

I ran the build.py script and then ran build_tests.sh.bat and didn't get any errors.

Then I compile with the following:

gcc -g -Wall -Wextra -std=c11 -Izpl/code -o a.out test.c -pthread -lm -ldl

and get the following errors:

zpl/code/zpl.h: In function ‘zpl_vm_alloc’:
zpl/code/zpl.h:7450:23: error: ‘MAP_ANON’ undeclared (first use in this function); did you mean ‘FP_NAN’?
 #define MAP_ANONYMOUS MAP_ANON
                       ^

zpl/code/zpl.h:7480:41: error: ‘MADV_DONTNEED’ undeclared (first use in this function)
     int err = madvise(vm.data, vm.size, MADV_DONTNEED);
                                         ^~~~~~~~~~~~~

zpl/code/zpl.h: In function ‘zpl_fs_get_type’:
zpl/code/zpl.h:11157:25: error: ‘S_IFDIR’ undeclared (first use in this function); did you mean ‘S_ISDIR’?
         if( s.st_mode & S_IFDIR )
                         ^~~~~~~
                         S_ISDIR

test.c

#include  <stdio.h>
#define ZPL_IMPLEMENTATION
#include "zpl.h"

int main (int argc, char** argv)
{
	zpl_file_contents fc = zpl_file_read_contents(zpl_heap(), true, argv[1]);
	zpl_json_object root = {0};
	u8 err;

	zpl_json_parse(&root, fc.size, (char *)fc.data, zpl_heap(), true, &err);
	printf("%s\n %s\n", root.name, root.string);
	zpl_json_free(&root);
	zpl_file_free(&fc);

	return 0;
}

Not sure if there is an issue in zpl or if I'm doing something wrong.
Thanks!

`zpl_jobs_process` is expensive

zpl_jobs_process, usually called in a loop, has at least two expensive operations:

• sort array by priority jobs.c#L141
• remove array element at index 0 jobs.c#L155

Fix inline methods to be truly inline

Many zpl methods are supposed to be inlined, however their definition is only included in a file that includes implementation part in particular.

This might not be an issue for SCU (unity) builds, however proves to be problematic when multiple compilation units are involved. CUs that only include the declaration part of ZPL will fail to inline methods and will fallback to being resolved by referencing external symbols from the single CU including ZPL implementation.

To solve this issue, build system should ideally make sure inlined methods are always available when including ZPL (without using ZPL_IMPLEMENTATION). The solution should cause minimal damage and be invisible to the end-user. Ideally, this fix should be automated by the build system.

New iteration

I've decided to review and refactor these libraries to improve project's health and maintenance effectivity.
These are concerns I would like to discuss:

Module naming

I would propose to shorten the module name prefix to zpl[2-4 char re-presentation] from our current zpl_. E.g. zpl_cull -> zplcull, zpl_image -> zpli.

Motivation

Names get too long where some modules are hard to use and hard to read. Having shorter names would improve readability and thus productivity.

Documentation

I would like to ask for an act of documenting our modules and how to exactly do that. I would like to discuss possible and viable ways to make these libraries easier to use.

Motivation

Undocumented parts of the libraries are harder to understand from the code and require a working example on their usage.

Version standard

I would like to propose using http://semver.org/ as our versioning standard and reset the current versions. From now on, PRs have to update version on the last commit. E.g. if you push 12 commits, collate all changes to 1 version and update accordingly.

Motivation

The current version naming is inconsistent and confusing. Updates are often breaking the API and make it harder to understand what changes were made.

--
Feel free to discuss and vote on possible options and ideas.