I am just some random dude that cares quite a bit about how to develop great and useable software (which is pretty hard!). Mostly interested in Java / Kotlin, but C# isn't that far off :)
Feel free to contact me about anything!
datus enables you to define a conversion process between two data structures in a fluent functional API
License: MIT License
There are numerous scenarios where an output object is constructed out of multiple input objects. datus does not support this concept as it has severe implications:
convert(Collection<In1>, Collection<In2>)
as it does not correlate the two collections, we would need a map then, which seems cumbersomeMaybe datus should define Tuple2...X types, these would easily fix this problem by unifying multiple input types into one, though the user has to create them by hand. Great care should be taken when deciding this as I don't want to implement too many concepts that are not directly related to data-mapping.
As a library datus should follow PECS. Most parts of datus are already PECS compliant but some functions that were added just before releasing 1.0 don't comply yet (or they have just slipped my attention).
I will categorize this issue as semver-minor because applying PECS is backwards compatible and not being able to use functions/predicates/etc. that would work with PECS is more of a bug than a new feature.
Mapper
interfaceHello there!
My name is Ana. I noted that you use the mutation testing tool in the project.
I am a postdoctoral researcher at the University of Seville (Spain), and my colleagues and I are studying how mutation testing tools are used in practice. With this aim in mind, we have analysed over 3,500 public GitHub repositories using mutation testing tools, including yours! This work has recently been published in a journal paper available at https://link.springer.com/content/pdf/10.1007/s10664-022-10177-8.pdf.
To complete this study, we are asking for your help to understand better how mutation testing is used in practice, please! We would be extremely grateful if you could contribute to this study by answering a brief survey of 21 simple questions (no more than 6 minutes). This is the link to the questionnaire https://forms.gle/FvXNrimWAsJYC1zB9.
Drop me an e-mail if you have any questions or comments ([email protected]). Thank you very much in advance!!
This tickets serves as the central point for anything related to datus documentation that could be improved.
Let me know which parts you would like to see improved :)
A user of datus has noted that .from(Function.identity()
feels clumsy when a mapping step has to combine multiple getters of the input type into one setter / parameter of the output type.
We should consider adding a shortcut. Naming seems to be the hardest part of this task as the implementation is pretty straightforward.
The USAGE.md file is pretty bare and an in depth explanation of the different constructs of datus is required to ease the usage of datus.
I am a little confused about PerformanceBenchmarkTest
.
.jvmArgs("-server")
is an ancient setting, that is simply ignored in the JVM.
Second, what is the point of having a single @Benchmark
? You are supposed to benchmark against something else, I guess.
It seems your testedConversionShouldBeCorrect
, just ensures that the conversion is correct, but that is something JMH already offers via @TearDown
: https://hg.openjdk.java.net/code-tools/jmh/file/b6f87aa2a687/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_05_StateFixtures.java
Anyway, these are nitpicks, mainly. The idea is that I hoped this test would have some samples against some popular frameworks. I have seen https://github.com/arey/java-object-mapper-benchmark, though. It's just that your results rely on those tests and I am not sure about their correctness (without understanding them, those are meaningless numbers: look how the benchmark presents that "direct" mapping is slower than "map-struct"). I wonder if I should provide a PR against some data mappings libraries inside (or a separate project/child from datus)?
Currently the Datus
class expects two Class<T>
parameters to infer the input and output types. These cannot be passed if e.g. T
is a List<T>
as List<T>.class
cannot be created / obtained.
We should add an overload to Datus.forTypes()
without any parameters that leaves the type definition to the caller via Datus.<List<Integer>, List<String>>forTypes()
(it is ugly indeed, but it works, don't think there is a better way).
Thanks to #33 for noticing this issue ! :)
The JavaDoc does not state what happens when there is a key-collision for a given input collection. We should clarify that the last mapped value is retained in the result map and any other behaviour should be implemented via the conversionStream function which offers all other alternatives through the Collectors.toMap collector
When is a reserved keyword in Kotlin which forces Kotlin users to use the when function of the builder APIs with backticks - annoying.
Goal of this issue is to document any other approaches to the immutable API which offer a better usablity than the current implementation.
The following parts of the immutable API come to mind:
map
vs mapTo
.to(Function.identity())
I am open for suggestions but releasing 1.0 does not necessitate a new immutable API.
EDIT: solution was found, see below
Currently datus cannot be used in Kotlin as outlined here - Kotlins compiler can't infer the generic type parameters as accurately as Java.
We should add some tests for Scala and Kotlin (once the inference is more capable) to ensure datus really works with these languages.
When using orElse(null)
the java compiler complains about an ambiguous method call (supplier variant and value variant are equally specific) and some user might prefer orElse(null)
over orElse(Function.identity())
when doing a null check.
We should add another orElse
variant that uses a null value directly, e.g. orElseNull()
- naming should be straight forward.
Suppose I have two DTOs:
public class In {
public String getAge() {
return age;
}
public void setAge(String age) {
this.age = age;
}
private String age;
}
and
public class Out {
public Integer getAge() {
return age;
}
public void setAge(Integer age) {
this.age = age;
}
private Integer age;
}
And I want to conditionally apply a transformation: I want to map String age
only if it is made from digits, otherwise simple don't map it.
The only way I found to do this is:
In in = new In();
in.setAge("no");
Out out = Datus.forTypes(In.class, Out.class)
.mutable(Out::new)
.from(In::getAge).given(Pattern.compile("\\d+").asPredicate(), Function.identity())
.orElseNull().map(x -> x == null ? null : Integer.parseInt(x)).into(Out::setAge)
.build()
.convert(in);
Assertions.assertNotNull(out);
Assertions.assertNull(out.getAge());
which looks weird, at best. Did I miss some documentation and there is a better way? If no, I think I might submit a PR with my idea to handle this; unless you already have something like this in motion.
datus makes extensive use of lambda-chaining (.map
-steps) and wrapping other lambdas (e.g. to support implicit nullsafeness). The resulting wrapper-functions share their byte-code / method-body with all calls to these functions (multiple .map
-steps) which makes them mega-morphic and thus un-inlineable.
This affects the following places:
map
in both APIsnullsafe()
in both APIsI have asked on SO on how to solve this and Holger supplied a solution which improves datus performance in the benchmark suite from roughly 8 million op/s to 23 million op/s.
The following points need to be considered while implementing this feature:
Just leaving this as an issue to see if anyone is interested in a Kotlin port / compatability variant of Datus - leave a comment / reaction.
Hi,
What if i want to convert a property from source byte to destination string this future is currently supported by map struct.
Don't know how it slipped through, lets fix it ! :)
As much as datus is about explicitness: traversing nested objects which can be null is quite cumbersome as it stands.
We have to find a way to avoid something like this:
.from(Order::getCustomer)
.given(Objects::nonNull, Customer::getBillingAddress).orElseNull()
.given(Objects::nonNull, Address::getCity).orElseNull()
.into(OrderDTO::setBillingCity)
I personally would like to veto any variant that would make every mapping step of a mapping definition nullsafe - it's too implicit.
The existing documentation (readme.md & usage.md) may be enough but I myself like to play around with code (that compiles out of the box) instead of reading about a library. Let's add a simple example project that highlights datus features.
note: as this won't affect any release related code I don't know how to handle semantic versioning here - as it stands I won't do a new release after finishing the sample project.
This seems necessary as I have witnessed plenty of complex objects that far exceeded 8 constructor parameters
I reconsidered adding this functionality to datus as there are quite a few scenarios wherin simple data structures are recursively included in themselfs (parent child relations like categories or products containing more subproduct variants).
There is an utility class outlined in https://github.com/roookeee/datus/blob/master/USAGE.md#advanced-usage--faq to fix this problem. Implementing the given class once (+ unit tests) is far superior to letting users handle it themselves.
Let's add it :)
As its not possible to reference the currently being built mapper it is impossible to convert self-recursive data-structures (e.g. trees, double-linked-lists).
Say you have a Customer
object as a field inside your Order
object and want to extract both the cutomers name and adress to two separate fields of the output object. With datus you have to do getCustomer()
twice which is cluttered and quite cumbersome.
See this benchmark suite with the following manual code:
public final class ManualMapper implements OrderMapper {
@Override
public OrderDTO map(Order source) {
OrderDTO target = new OrderDTO();
Customer customer = source.getCustomer();
if (customer != null) {
target.setCustomerName(customer.getName());
if (customer.getBillingAddress() != null) {
target.setBillingCity(customer.getBillingAddress().getCity());
target.setBillingStreetAddress(customer.getBillingAddress().getStreet());
}
if (customer.getShippingAddress() != null) {
target.setShippingCity(customer.getShippingAddress().getCity());
target.setShippingStreetAddress(customer.getShippingAddress().getStreet());
}
}
if (source.getProducts() != null) {
List<ProductDTO> targetProducts = new ArrayList<ProductDTO>(source.getProducts().size());
for (Product product : source.getProducts()) {
targetProducts.add(new ProductDTO(product.getName()));
}
target.setProducts(targetProducts);
}
return target;
}
}
which currently looks like this:
private static final Mapper<Product, ProductDTO> productMapper = Datus.forTypes(Product.class, ProductDTO.class)
.immutable((String name) -> new ProductDTO(name))
.from(Product::getName).to(ConstructorParameter::bind)
.build();
private static final Mapper<Order, OrderDTO> orderMapper = Datus.forTypes(Order.class, OrderDTO.class)
.mutable(OrderDTO::new)
.from(Order::getCustomer)
.given(Objects::nonNull, Customer::getName).orElseNull()
.into(OrderDTO::setCustomerName)
.from(Order::getCustomer)
.given(Objects::nonNull, Customer::getBillingAddress).orElseNull()
.given(Objects::nonNull, Address::getCity).orElseNull()
.into(OrderDTO::setBillingCity)
.from(Order::getCustomer)
.given(Objects::nonNull, Customer::getBillingAddress).orElseNull()
.given(Objects::nonNull, Address::getStreet).orElseNull()
.into(OrderDTO::setBillingStreetAddress)
.from(Order::getCustomer)
.given(Objects::nonNull, Customer::getShippingAddress).orElseNull()
.given(Objects::nonNull, Address::getCity).orElseNull()
.into(OrderDTO::setShippingCity)
.from(Order::getCustomer)
.given(Objects::nonNull, Customer::getShippingAddress).orElseNull()
.given(Objects::nonNull, Address::getStreet).orElseNull()
.into(OrderDTO::setShippingStreetAddress)
.from(Order::getProducts)
.given(Objects::nonNull, productMapper::convert).orElseNull()
.into(OrderDTO::setProducts)
.build();
We have to do something about this and null checking is a problem too :/
I just wanted to tell you that compared to some libraries (even my code, but I am not even ready for a release yet, so I might improve more), your code does a pretty good job. Here are my results run against github actions with latest JMH: https://github.com/wind57/0xMapper/pull/8/checks?check_run_id=820811700
You need to download the artifacts to see them.
You can close this issue or somehow reference to those, if you want.
Some of the unit-tests are rather odd and should be reconsidered / cleaned up to be more understandable
datus is subjectively feature complete but feature request / maintenance is still done whenever needed
It can be painful to map all the fields. Why do we need to map them even when the names are the same?
see title
datus was just integrated here but the main repository could use some fancy images + benchmark statistics itself. The benchmark should focus on other mapping frameworks that don't use code generation but we should definitely mention those that do and how they compare to datus perf
datus currently implements the Mapper<A,B>
in two distinct classes for the mutable and immutable API. This is unnecessary as the Mapper<A,B>
interface is a functional interface. As the mapper implementation classes are package private this is a non-breaking change that reduces the overall code size of datus while also opening up for future JVM optimizations in regards to lambdas. Some local JMH testing also showed minor performance benefits when using lambdas.
<self explanatory>
Could take some time to be succinct and as unbiased as possible
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.