Giter Site home page Giter Site logo

Comments (31)

ckormanyos avatar ckormanyos commented on May 19, 2024 1

a good plan

Thanks John. OK @johnmcfarlane I took some initial steps, essentially those having no impact on current library use.

  • The test/example headers which are, in fact, solely needed for testing have been removed from the main public directory.
  • The utility header file <util/utility/util_dynamic_array.h> and its former directory have been removed.
  • uintwide_t is now single-header only, dependency-free.

Namespaces and header uintwide_t.h location, however, remain unchanged. The latter would be easier to address in a future change. For the moment, I am a bit reticent to change the existing namespace(s).

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024 1

I pretty-much integrated wide-integer into CNL

That is great! Thanks for pushing forward on this, John.

It's not a totally trivial change though. There's be a few places in your header where you assume math is global scope. Those'll need fixing.

Thoughts?

I would like to try. I'm wrapping up the clang-tidy checks and I would like to have the bulk of these finished before moving on to the architectural changes.

I like your idea --- in particular the one presented here --- because it allows us to move forward with CNL integration while taking a small-ish step forward in the present architecture.

Let me see how the clang-tidy stuff pans out... Then we can move forward on this idea (or similar forward method as it progresses).

Thank you @johnmcfarlane!

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

the following observations/suggestions

Thanks for these interesting inputs, John. I think we can find some common compromises, but we need to discuss these just a bit.

Some of these ideas I have also had in similar form(s). These kinds of chasnges could potentially percolate also into my embedded projects extend to some of my other numeric types as well.

I believe the namespace change might be a rather small but breaking change. In the past, I have announced these kinds of breaking changes about a month up front and then implemented them, with no problem really after such things.

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

Let me wrap my mind around these suggestions and soon get back to you. Then we can decide if the PR is the right place to draft a version adhering to these suggestions, as i agree it probably will be. Then we can see how a working model, let's say, plays out.

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

@ckormanyos no problem. I can work with the multiple directories, headers, namespaces with straight-forward workarounds for now, so no hurry. If CNL comes to depend on wide-integer, they will end up being installed on systems together wherever CNL goes but that's a ways off.

A thought in that direction: while the option of absorbing the algorithms into CNL (as I think you mentioned in another thread) would avoid a lot of complexity, I'm interested in exploring paths which involve maintaining links between the various projects so that fixes and improvements which benefit one project benefit them all.

The application of a build system like CMake and a package system like Conan to a project is one way of making it less painful to plug projects together. Particularly if you're using util_dynamic_array.h in other projects, it's worth knowing that there are ways to reuse this work in a more automated way.

See this PR for latest progress on tieing everything together. This branch is a loooong way from merging!

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

Well I kind of have to do something. When I first started collecting a few numeric types (independently from my work in Boost), I found myself gathering algorithms and code sequences from as alo as last century.

At first, I put some of them together in rather coherent, yet independent, forms. I now have several nice numeric types (with maybe a few more to come over the years) and several key projects (both published as well as unpublished) that use them.

But it's getting silly with redundancies. I believe my wide-decimal header is used by myself in at least 3 redundant published copies. Also wide-integer is on the same redundant path. Redundancy can be good, it has the advantage that you get only what you need, lightweight, independent and non-dependent on some kind of UNIX philosophy or similar. The disadvantage is redundancy itself, multiple copies, synchronization woes, error propagation and the like.

So I will indeed be working on reducing redundancies and finding more let's say standard storage/distro modes. The tried and true *nix usr/include and urs/lib is a classic --- with local instances of /include and /lib for a given project.

I am a bit stuck on the namespace. I like having the namespace math. But this has come into strong conflict with the popular namespace boost::math so I often end up needing to use my own ::math namespace being resolved with the scope resolution operator. This is kind of inconvenient.

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

Unfortunately, math is not a good choice for the reason you mention and the ones I gave above: just because it's maths-related doesn't lead to wanting it in a "math" namespace. Namespaces are not for taxonomy. It would be better to think up something that's a proper noun.

I started CNL with the name fixed_point, which fails to differentiate it from all the other fixed-point projects. Similarly, I know of a few wide integer projects and I'm sure there are others. By choosing an arbitrary name (like Hana, Clara, Spirit or Boost) or an acronym (like CNL or CTRE) you reduce the likelihood of name collisions (the main reason you'd ever want to use a namespace) and increase usefulness of the name for users.

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

Hi @johnmcfarlane a very first level simplification could:

  • Remove uintwide_t_examples.h and uintwide_t_test.h and place these in the relevant text or examples folder(s).
  • Integrate util/utility/util_dynamic_array directly as a file-local detail. This project has no need for a separate utility folder.

I do not think this would influence existing clients of wide-integer. This might not even really help your points in this issue, but does this help?

The potentially ambigous namespace(s) would remain, but single-header only would be the benefit...

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

That sounds like a good plan. TBH I haven't looked at the integration since I started to write my error handling document. I hope to turn back to it soon.

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

OK, I'm up to speed with the changes. We can probably resume this conversation. Much has been addressed since the original post. Would you like me to start a new issue with outstanding suggestions or maybe strike through what no longer applies?

Another item I don't think I mentioned is that _t is generally reserved for typedefs. As alluded to in the CompilerExplorer example here, my recommendation would be for two types: wide_integer and integer. These are the latest suggestions from the committee. Certainly uintwide_t now has a misleading u at the start and closely resembles the typedefs from <cstdint> which isn't what that type is about. If they remained a single type, perhaps multiword_integer or something that calls out the multiple limbs.

What are your latest thoughts?

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

OK, I'm up to speed with the changes. We can probably resume this conversation. Much has been addressed since the original post. Would you like me to start a new issue with outstanding suggestions or maybe strike through what no longer applies?

Yes. I believe we have done well moving forward. Thanks for your help.

Another item I don't think I mentioned is that _t is generally reserved for typedefs. As alluded to in the CompilerExplorer example here, my recommendation would be for two types: wide_integer and integer. These are the latest suggestions from the committee.

Certainly uintwide_t now has a misleading u at the start and closely resembles the typedefs from <cstdint> which isn't what that type is about. If they remained a single type, perhaps multiword_integer or something that calls out the multiple limbs.

What are your latest thoughts?

My unfiltered gut-reaction is to leave the preliminary work of uintwide_t closely as-it-is. A lot of people are using it. Big name changes and redesings of this scale might almost be better done in a new, mostly-independent derivative work that more closely parallels the progress in SG6. I'm just not sure if I would like to metamorphosize uintwide_t --- with all its attributes --- to try to adhere to where we it's going in the committee.

In order to at least add some conformant-look to uintwide_t, we could add some (potentially templated) typedefs that more closely resemble wide_integer and integer.

But if you ask my honest opinion, I feel like the scope of changes being discussed at this point almost need a new derivative work that is more closely resembling a partial reference implementation for wide_integer and integer, which might heavily borrow code or gain inspiration from ckormanyos/wide-integer.

I'm pretty sure the committe needs a reference implementation when dealing with numerics on this complexity level for sure. I would need and expect one. Now just one small part would/could be types like those in my project. I think the drive forward can and should be more closely in line with progress in SG6 --- fully knwoing, ... that is or would be a lot of work.

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

My unfiltered gut-reaction is to leave the preliminary work of uintwide_t closely as-it-is. A lot of people are using it. Big name changes and redesings of this scale might almost be better done in a new, mostly-independent derivative work that more closely parallels the progress in SG6. I'm just not sure if I would like to metamorphosize uintwide_t --- with all its attributes --- to try to adhere to where we it's going in the committee.

If ckormanyos/wide-integer formed the basis for a new type or library, how would support for that work? Would it be a snapshot of wide-integer which was then maintained by me, or by both of us? And would you wish to continue supporting / advancing wide-integer separately?

From CNL's PoV, what happens to ckormanyos/wide-integer is not as important as ensuring we're not expending duplicate effort and that CNL can continue to benefit from the expertise and effort you put into numerics. If I fork wide-integer or snapshot it inside CNL and then wide-integer continues to enjoy bug fixes and improvements from you, then I've created the burden of porting those over. That's really the thing I'm trying to avoid here.

In order to at least add some conformant-look to uintwide_t, we could add some (potentially templated) typedefs that more closely resemble wide_integer and integer.

That would work well except that anyone using those aliases (including CNL users) would also get ::math and ::util added into their global namespace. We could flip your idea on its head and turn uintwide_t into an alias of those new types:

#include <ckormanyos/integer.h>
#include <ckormanyos/wide_integer.h>

namespace math::wide_integer {
  template<const size_t Width2,
           typename LimbType,
           typename AllocatorType,
           const bool IsSigned>
  using uintwide_t = std::conditional_t<
      std::is_same_v<AllocatorType, void>, 
      ::ckormanyos::wide_integer<std::array<LimbType, (Width2 + width_v<LimbType> - 1) / width_v<LimbType>>>, 
      ::ckormanyos::integer<LimbType, std::vector<LimbType, AllocatorType>>>;
}

In this way, wide-integer should remain backward compatible but it would be drawing on the new library. (Note I think I could reduce the number of wide_integer template parameters.)

This is a good choice if you want to keep uintwide_t in active development. But if you're happy to draw a line under it (call it V1) and move on to a new, incompatible interface (call it V2), then it's not necessary because uintwide_t stops being actively worked on.

It's up to you.

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

If ckormanyos/wide-integer formed the basis for a new type or library, how would support for that work? Would it be a snapshot of wide-integer which was then maintained by me, or by both of us? And would you wish to continue supporting / advancing wide-integer separately?

From CNL's PoV, what happens to ckormanyos/wide-integer is not as important as ensuring we're not expending duplicate effort and that CNL can continue to benefit from the expertise and effort you put into numerics. If I fork wide-integer or snapshot it inside CNL and then wide-integer continues to enjoy bug fixes and improvements from you, then I've created the burden of porting those over. That's really the thing I'm trying to avoid here.

Good points, John. That is, in fact, what I was struggling with in my own thoughts. I thought, ughhh new project, new problems, old project vague support. In these senses it really would make sense to stay consistent with the existing work and evlolve it.

We could flip your idea on its head and turn uintwide_t into an alias of those new types.

Ah yes, of course, that's it. Very stable. Great proposal @johnmcfarlane.

It would avoid breaking change(s), and simultaneously allow for classes of larger/different/more-conformant nature to be built up while fully and actively/dynamically/in-living-project retaining the legacy uintwide_t support if community needs it. I hadn't thought that through yet. Agreed. We should move forward on that line.

We need to take a bit of time, summarize the remaining points of this issue, include the legacy uintwide_t` support and implement the compatibility types --- moving forward.

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

Sounds good. And yes, it won't be trivial, but I think the results will be something many users will be eager to adopt.

For reference, CNL v2 (which is yet to be release) will be a breaking release. That's why I am bumping the big number. I'm sure many users continue to use v1 - if for no other reason then because it supports much older compilers and standards. At such time that they decide to move to C++20, they can then consider moving to v2 but v1 has only fairly minor defects that I know of. This might be an upgrade path to consider offering your users also. (Note that despite the terse C++20 code I'm using to illustrate the proposal, I am not pushing for you to drop C++11 support!)

I've updated the proof of concept here. Note there are fewer template parameters: once you know the container type, then you already know the size/limbs.

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

Sounds good. And yes, it won't be trivial, but I think the results will be something many users will be eager to adopt.

Yep. Agreed. I've wrapped my mind around it now.

I've updated the proof of concept... Note there are fewer template parameters: once you know the container type, then you already know the size/limbs.

Yes. Great. Let's work together --- in the sense that we have already been doing, or if you like, share the coding even more (all at your option, of course)? There is some significant coding for the first evolutionary step and will take some time.

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

Let's work together --- in the sense that we have already been doing, or if you like, share the coding even more (all at your option, of course)? There is some significant coding for the first evolutionary step and will take some time.

Yes, I recommend small incremental changes to get from where we are to the end state.

First we need to decide what that end-state is. How - if anything - does it differ from the proof of concept? And what are the locations of the files? There is a convention of putting public headers under an /include/ directory (example1, example2). Would the new interface include backward compatibility to uintwide_t? That's a given. Next question, does new API live in ckormanyos/wide-integer repo?

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

recommend small incremental changes

Yes. For that, I'd like to resolve a question on internal details. If we have a generalized template storage (maybe templated), then do we require internal container storage to have random-access iterators complete with operator[](size_t) on the container members? If so, then the template container choices are limited. If random-access is not required, then I would have to change some internals that use random-access operator[](size_t) to access the values of the member storage variable values.

These would be simple changes, but i'd do these first within the context of uintwide_t before larger desigh refactoring.

Thoughts on container to have random-access or open with only iterator support required?

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

tl;dr probably yes, it requires random access if you're using operator[] but that's no big deal.

It's a good question. You're really the one to answer that because it essentially depends on the algorithms performed over the elements currently.

There's no problem requiring that the container support random access. It is, in fact, tricky to express this in code before C++20 but that's generally not considered too bad a problem: most likely, if the container doesn't have an operator[] then compilation will fail with a somewhat unhelpful error message.

The fact that C++11 is supported doesn't really change this situation. But it does mean that something other than std::array needs to be provided if the user wants constant expressions: std::array does not have good constexpr support before C++20.

Irrespective of this, whenever you can express algorithms in terms of iterators, it's usually a worthwhile abstraction to aim for. It may even help the compiler decide where it can optimise. For example, you could express operator^ in terms of std::transform:

auto operator^(lhs, rhs)
{
  assert(lhs.storage.size() == rhs.storage.size()); // because this is a limited example - not because you cannot do ^ for different width numbers
  Container result(StorageType{lhs.storage.size()});
  transform(begin(lhs.storage), end(lhs.storage), begin(rhs.storage), begin(result), [](auto l, auto r) { return l^r });
  return MyNumberType{result};  // assuming there's a constructor
}

Now, if you're using this approach, that's great... except std::transform isn't constexpr either! IOW, while you're encouraged to avoid [], it's not essential to this refactor and you're up against it anyway because of legacy support.

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

while you're encouraged to avoid [], it's not essential to this refactor.

I'm giving it a try in a branch --- that means giving it a try to avoid operator[](...) --- and it seems harmless enough with a hundred or two hundred trivial changes.

I'll incubate that and see if it retains speed/passes-tests/embedded-friendly, etc. If that incubates, I'll probably go with iterator-internals, since that's actually the better way to generic programming in the long term.

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

Great! However, this can be pursued in isolation from the kinds of refactors covered by this thread.

At the end of, you may find you can make numbers out of std::list. That would be a gas! 😆

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

may find you can make numbers out of std::list

That's the plan, maybe not std::list, but I do have some very specialized SRAM-containers and off-chip peripheral containers from the embedded world that would actually greatly benefit from the template container type. Since I want generics here, I decided to push through with the removal of uses of operator[]() random access in the internals. It plays well in the implementation and is in main branch.

updated the proof of concept here

Ah yes. That looks nice. I can see the Boolean parameter. I hope the container of limbs can remain private in the base class.

In the first draft, I might reduce the scope of design to the bounded case and see how that goes. The class design in the proof of concept looks sensible. I will attempt to make a draft and see how it plays. Thanks John!

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

YW. Don't be afraid to add an accessor akin to vector::data():

constexpr Container& data() noexcept;
constexpr Container const& data() const noexcept;

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

Happy New Year! Update: I pretty-much integrated wide-integer into CNL by copy-pasting uintwide_t.h into the CNL repo. Then I've mulled over it. I'm still not comfortable with introducing ::math to my users' programs but I think there's a workaround.

An optional pre-processor macro, WIDE_INTEGER_NAMESPACE, which, if defined wraps all the definitions in

namespace WIDE_INTEGER_NAMESPACE {
...
}

I can then include like so:

#define WIDE_INTEGER_NAMESPACE cnl::_impl
#include "_impl/uintwide_t.h"

It's not a totally trivial change though. There's be a few places in your header where you assume math is global scope. Those'll need fixing.

Thoughts?

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

'm happy to continue working on the namespace change. I need to test that it works with CNL anyways.

Yes, please feel free to forge ahead on that endeavor, John.

Just a head's up... You might see that a rather big refactor is underway for clang-tidy syntax changes. See also #143 and more recently #145.

I don't expect that these cause large merge conflicts, as it's syntax only. But there might be some merging needed depending on when you branch or synchronize things. i will try to wrap up the clang-tidy bulk of the work ASAP.

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

Cool. I've thrown together a PR in #163. LMK what you think.

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

Hi John (@johnmcfarlane) I am returning to this older issue in order to rekindle our discussion and see what's already done and what next potential steps (if any) could help.

In this thread, we discussed various topics regarding;

  1. outer namespace,
  2. favoring iterators for internal algorithms,
  3. expressing the input template parameter(s) via generic container,
  4. anything else I/we missed that could do any possible good for standardization, etc.

Numbers 1 and 2 are finished. I would like to make progress on number 3 and potentially number 4.

Personally, I would like to handle nubmer 3 (potentially in a derivative work that uses uintwide_t. In particular, i would like to use a storage container based on external memory silicon-chips for tiny embedded systems. These kinds of things are in my book on microcontroller programming and having off-chip big integers would be cool for my own future developments in various fields.

Regarding number 4, I do not know if there is anything on your mind that could help, or if synergy is even expected to help. In this matter, I'm happy with wide-integer in its current state. The community seems to be using it and some developers have contributed change requests for new functions that have been done. If, however, there are a few easy adaptions in your mind, we could discuss these in further issues.

Thoughts?

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

Thanks for your sage input, as always, John!

external memory silicon-chips sounds fascinating but I'm not familiar with that technology. Do you think the iterator approach would help bridge the g[ap] with conventional memory representations? If so, I'd recommend exploring that

Yes. In fact, I would like to make specialized containers that reside in external memory (off-chip) or also reside within FPGA-memory zones for hardware-acceleration in the future. This is a research topic that interests me. I have previously made memory chunks and std::array-like containers in program-code of certain microcontroller ROM-areas, etc., but not fully made containers that reside in a different random-access physical space (or computational zone) yet. I'll be looking into this, also for faster accelerations, as I feel this topic has not yet been fully handled. This field could work in all kinds of areas, it just happens that wide-integers are very useful for proof-of-concept of such things since an exact, known answer is intuitive and easy-to-get from integral calculations.

regarding CNL integration, I think we're at a good place.

This is great.

so it wouldn't make sense to be deviating wide_int from it's current course

OK. I did actually go entirely in a prototyping run for replacing the internal storage in wide-integer with std::list<limb_type>. This was initially a bit difficult since list does not have random access iteration and my algorithms had, in fact, required index addition/subtraction beyond incrementation. I will probably actually finish a version which is capable of handling internal storage and algorithms with any generic bidirectional iterator (as opposed to requiring random-access-iteration), as long as no performance in the simple random-iteration case is lost. I'll summarize this point more below.

At this stage, I might make one final adaption of the work wich replaces today's AllocatorType parameter with a ContainerType, whereby type definitions (i.e., aliases) maintain the compatibility with the existing code base. I might also elect to not do this. I have not yet decided.

Re. 4, If you're happy with it, I would proceed as is. I'm pleased to hear it is getting some use it there. And given the way it is integrated into CNL, anyone using the more complex types is already benefiting from it there too.

I'm really happy to hear that. So all-in-all, we could actually let this issue settle, possibly close it.

In summary, I will keep today's wide-integer interface as a supported feature. I might adapt to a generic container approach, and I probably will make all internal algorithms bidirectional-iterator-compatible.

The latter point (making all internal algorithms in wide-integer to be bidirectional-iterator-compatible) should be expressed as a separate issue. I'm actually almost done with this in a branch.

from wide-integer.

johnmcfarlane avatar johnmcfarlane commented on May 19, 2024

from wide-integer.

ckormanyos avatar ckormanyos commented on May 19, 2024

choice of list was mostly to demonstrate genericity, rather than for efficiency!

Oh yes, absolutely. The experience using std::list revealed that I had used random-access iteration in nearly 200 (more than I had thought!) places within the thousands of lines in wide-integer. I thus did a rather massive refactor of these to handle bidirectional iteration.

At the end of refactoring I had not lost any performance for random-access iterators. So that was cool. This work does, however, need to settle for a while and I still need to optimize some more areas for the generic case.

from wide-integer.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.