virgesmith/humanleague

Microsynthesis using quasirandom sampling and/or IPF

humanleague's Issues

Allow marginal oversampling

info about this package

Hello.

I have found this package by chance on github, and I am very interested!

I am currently using the mipfp package in R to synthetise a populations used later on for modelling purposes in Python (package here). I was looking for a semi-replica of this package in Python, to have everything in one place, however I am not sure if the multidimensional IPF is integrated, and I can't find any example/documentation showing an example on how to use the package in Python.
Can you please give advice? help in some way?

best wishes and many thanks

Implement IPF-based quasirandom sampling

node.js API

PyPI / conda-forge submission

Current PyPI package is broken. Move from distutils to setuptools and implement workaround for numpy dependency at build stage.

Further testing/development/generalisation of constrained and correlated sampling

Potentially generalise and allow >2 dimensions?
Test algorithm performance: unconstrained sampling+adjustment vs continual recalculation of sampling probabilities
Unify exogenous constraint/correlation into a probability matrix? (esp. given correlation isnt very meaningful here)

Increase number of unit tests drastically

use pybind11

sobolSequence segfault

if dimension input > 1111. trap this

$ python -c "import humanleague as hl; print(hl.sobolSequence(1112,2))"
Segmentation fault (core dumped)

sequence length of zero also segfaults

Create 1.0.0 branch

.. to allow major API-breaking work on next version

gcc fix

Dear Maintainer,

could you please fix your package - simply add missing include directive as in the patch - to build on gcc13 (on Windows)? This is needed for the package to work with upcoming version of Rtools.

Thanks
Tomas

Binary files orig/humanleague/src/.NDArray.h.swp and patched/humanleague/src/.NDArray.h.swp differ
diff -Nru orig/humanleague/src/NDArray.h patched/humanleague/src/NDArray.h
--- orig/humanleague/src/NDArray.h	2023-04-17 20:02:36.000000000 +0200
+++ patched/humanleague/src/NDArray.h	2024-02-15 08:29:31.546660700 +0100
@@ -9,6 +9,7 @@
 #include <cstddef>
 #endif
 #include <cassert>
+#include <cstdint>
 
 // The array storage
 template<typename T>

Implement "flatten" function in R interface

• consistent with python interface
• tabular output can use a lot of memory and may not be required - so implement separate function to convert multidim array output into table
• also provide column names

CRAN fix

Dear maintainer,

You have file 'humanleague/man/humanleague.Rd' with \docType{package},
likely intended as a package overview help file, but without the
appropriate PKGNAME-package \alias as per "Documenting packages" in
R-exts.

This seems to be the consequence of the breaking change

Using @doctype package no longer automatically adds a -package alias.
Instead document _PACKAGE to get all the defaults for package
documentation.

in roxygen2 7.0.0 (2019-11-12) having gone unnoticed, see
r-lib/roxygen2#1491.

As explained in the issue, to get the desired PKGNAME-package \alias
back, you should either change to the new approach and document the new
special sentinel

"_PACKAGE"

or manually add

@Aliases humanleague-package

if remaining with the old approach.

Please fix in your master sources as appropriate, and submit a fixed
version of your package within the next few months.

Best,
-k

Remove conda builds

See here:

https://conda-forge.org/docs/maintainer/updating_pkgs.html#archiving-feedstocks

then remove from CI and docs

Implement IPF

Add slack build notification

.

Multidimensional (labelled) marginals

Allowing microsynthesis of the form AB * AC -> ABC

QISI does not check seed array shape is consistent

permit a zero total pop in prob2IntFreq

to avoid having to special-case, e.g.

      else:
        # TODO permit zero total in prob2IntFreq to avoid this branch
        if occupants:
          occ_array = humanleague.prob2IntFreq(np.full(establishments, 1.0 / establishments), occupants)["freq"]
        else:
          occ_array = np.zeros(establishments)

1.0 fixes/enhancements as required by household_microsynth

should be merged to master

Greater control of sequence generator state?

work out best approach to this...

• make generators static (singletons) and provide reset API
• just make greater use of the skip argument (documenting that you can use this to generate different sequences)

IPF Result Integerisation Question

Hi!
Thank you very much for making this package public, it is a very useful tool.

I'm just getting into population synthesis and have a small question. I used the flatten() function to convert a multidimensional array to a dataframe after getting the results using the ipf(), but since the function can only convert integer arrays, integrating the results results results in a decrease in the overall population size.
I see you guys mentioned integerisation in your previous question, is this relevant? What should I do to solve this to get the same int number of total population?

Thank you so much!!

General Refactoring

Remove legacy functionality post 2.0 release

v2.1 release

Redesign API

allowing labelled multidimensional marginals (see #5)

Reinstatement on CRAN

Was removed as the email in DESCRIPTION no longer works. How to fix?

Investigate/understand why QISI has better degeneracy than QIS

QIS using solve_p should give be exactly the same results as QISI with unity seed

Sobol now available in scipy

Hi @virgesmith - you might be interested to know that a Sobol generator has been implemented in scipy. We're using this in the SALib library rather than our own implementation.

2.0 Release / JOSS & CRAN submission

Install on MacOSX raises gcc error

Collecting git+https://github.com/virgesmith/humanleague.git@master
  Cloning https://github.com/virgesmith/humanleague.git (to master) to /private/var/folders/mm/v_zfd9yx3r5bg0xsp0mxgqv80000gr/T/pip-rtn1jmmc-build
Installing collected packages: humanleague
  Running setup.py install for humanleague ... error
    Complete output from command ~/miniconda3/envs/testenv/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/mm/v_zfd9yx3r5bg0xsp0mxgqv80000gr/T/pip-rtn1jmmc-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/mm/v_zfd9yx3r5bg0xsp0mxgqv80000gr/T/pip-owh94c8v-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    building 'humanleague' extension
    creating build
    creating build/temp.macosx-10.9-x86_64-3.5
    creating build/temp.macosx-10.9-x86_64-3.5/src
    creating build/temp.macosx-10.9-x86_64-3.5/humanleague
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/will2/miniconda3/envs/testenv/include -arch x86_64 -DMAJOR_VERSION=2 -DMINOR_VERSION=0 -DPATCH_VERSION=0 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I. -I/usr/include -I/usr/local/include -I/Users/will2/miniconda3/envs/testenv/lib/python3.5/site-packages/numpy/core/include -I/Users/will2/miniconda3/envs/testenv/include/python3.5m -c src/Sobol.cpp -o build/temp.macosx-10.9-x86_64-3.5/src/Sobol.o -Wall -std=c++11
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/will2/miniconda3/envs/testenv/include -arch x86_64 -DMAJOR_VERSION=2 -DMINOR_VERSION=0 -DPATCH_VERSION=0 -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I. -I/usr/include -I/usr/local/include -I/Users/will2/miniconda3/envs/testenv/lib/python3.5/site-packages/numpy/core/include -I/Users/will2/miniconda3/envs/testenv/include/python3.5m -c src/SobolImpl.c -o build/temp.macosx-10.9-x86_64-3.5/src/SobolImpl.o -Wall -std=c++11
    error: invalid argument '-std=c++11' not allowed with 'C/ObjC'
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
Command "~/miniconda3/envs/testenv/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/mm/v_zfd9yx3r5bg0xsp0mxgqv80000gr/T/pip-rtn1jmmc-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/mm/v_zfd9yx3r5bg0xsp0mxgqv80000gr/T/pip-owh94c8v-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/mm/v_zfd9yx3r5bg0xsp0mxgqv80000gr/T/pip-rtn1jmmc-build/

Add multidimensional integerisation function

Input: n-D floating point matrix s.t. all 1-d partial sums are integer vectors
Internally construct marginals and perform QISI using input matrix as seed

Python API

Issues raised re: JOSS review and user support

• python requirements.txt missing numpy
• misleading documentation (ageByEthnicity array is actually ethnicityByGender)