kwazzi-jack/kal-cal

Kalman Filter and Smoother Implementation for Radio Interferometric Gains Calibration. This library is part of the master's work by Brian Welman and serves as a 'proof-of-concept' tool for it.

Hi, I'm Brian 👋

About Me 📖

Welcome to my GitHub! I am Brian, a bald, bearded South African postgraduate scholar with a deep fascination for Big Data, high performance and scientific computing. I am passionate about solving the scientific world's most challenging problems while absorbing any and all knowledge along the way. My academic journey is a tapestry of mathematics, radio astronomy, statistics and computer science.

Contact Me 📨

• 📧 Email: [email protected]
• 🔗 LinkedIn: /in/brianwelman
• 🌐 ORCID: 0000-0002-0254-5173
• 📸 Instagram: @brianwelman

Quick Facts ⚡

• 🔭 Previously worked on: Just finished my masters in radio interferometry, focusing on developing a fast, scalable and sophisticated prototype calibration framework called Kalman Filtering and Smoothing in Calibration (KalCal). My thesis describes the theoretical underpinnings of the framework and the excellent results from a simulated calibration experiment to justify further development.
• 🖥 Currently working on: A novel real-time or "on-line" calibration software suite, written in Python and Rust, called the On-line Radio Interferometric Calibration Suite (ORCa). It forms the basis of future doctoral research into unprecedented real-time calibration technologies for modern radio interferometers, which starts the path to a "world-first" radio interferometer image preview system.
• 🎓 Learning and Development: Mastering the Rust programming language and associated best software development practices. Additionally, I am advancing my Bayesian statistics knowledge to solve parallel and distributed computing limitations of Bayesian filters and smoothers.
• 👥 I’m looking to collaborate on: Any open-source projects! I love giving back; programming and problem-solving are fun in any form. My expertise lie in mathematical, numerical, statistical, Big Data, low-level/systems and scientific computing. I am also handy with optimisation, machine learning and AI, algorithm development, and theoretical issues.
• 🤔 Exploring: Currently, quantum computer science, LLMs and multi-modal machine learning models. In addition, I created open-source software aimed at assisting with mathematics writing and education. Finally, I am playing around with embedded systems programming and making chaotic microcontroller systems and machines!
• 💬 Ask me about: You can talk to me about most things, but I love discussing (and debating 😅) mathematics, computer science, physics, statistics and current trends within any of them. I also have a great deal of knowledge in the lore of Warcraft, Star Wars and Warhammer 40K, if you are keen to discuss!
• 📫 How to reach me: Best would be by email or LinkedIn. My eDoor is always open!
• Achievements: Besides my distinctions from my previous degrees, my favourite career achievement is deriving an alternate form of the Kalman filter equations logically identical to its original thing!

My Work 💼

Much of my work is centred around high-performance and scientific computing topics, especially in Big Data regimes, currently within radio interferometric calibration. I come from a pure mathematics and theoretical computer science background that has allowed me to optimise and solve large computation problems from both mathematical and computational fronts.

I am highly interested in the discovery, design and optimisation of algorithms, particularly in the context of:

• (Bayesian) Statistical Estimation Algorithms
• Self-Improving Models and Techniques
• Machine Learning Models
• Quantum Computing

Most of my work is centred around research and development. Still, the majority of it requires me to continuously learn and master new techniques and technologies to improve computing, particularly in science.

Featured Projects

Below is a list of my most notable projects thus far:

• Kalman Filtering and Smoothing in Calibration (KalCal): A calibration framework that uses the Kalman filter and smoother, alongside parameter-tuning techniques, to design and develop sophisticated radio interferometric calibration algorithms. The programming involved is distributed across several repositories.
• On-line Radio Interferometric Calibration Suite (ORCa): A new real-time or on-line calibration software consisting of the optimised computation server ORCa-Engine (Rust) and the web user-interface ORCa-UI (Python). The goal is to create a groundbreaking real-time calibration system using KalCal to improve current calibration strategies. This will be the start of creating a world-first radio interferometer preview system. It also tests the introduction and use of Rust within the Python, and C++ dominated radio astronomy software community.
• matque: A novel open-source software side-project that will generate mathematical questions and solutions for exercise, test or exam documents for various applications, e.g. homework, studying or formal tests. The goal is to make it highly customisable and diverse with natural language processing models while ensuring that solutions are 100% accurate (via a symbolic validation engine), reproducible and educationally insightful.
• wolf-whisper: An experimental open-source software that explores the ideas of creating software that translates human speech into LaTeX mathematics. It starts with OpenAI's whisper model for speech-to-text functionality, and it will end with a simple symbolic-to-tex using the sympy library. The current challenge is handling the middle layer that converts text-to-symbolic, e.g. "integral of x squared plus one" is integrate("x^2 + 1", "x"). An attempt was made via the Wolfram-Alpha API (hence the name), but the project has shifted towards exploring fine-tuned LLMs for this layer.

Skills

Research and Development Problem-Solving Public Speaking Teamwork Presenting Optimisation Big Data Technologies Mathematical Modelling Applied Research Mathematical Physics Algorithms Organisation Skills Statistical Modelling Statistical Analysis Machine Learning Data Visualisation Data Analysis Control Engineering Bayesian Statistics Bayesian Optimisation Automation Communication and Collaboration Object-Oriented Programming Functional Programming Software Development Software Design Debugging Virtualisation Databases Information Security Virtualisation

Technologies 🛠️

Rust Linux Python Shell Scripting LaTeX Docker Ubuntu/Debian Unix C/C++ Virtualisation

Fun Facts 🌟

• I love video games!
• I played rugby at a high school provincial level.
• I have a level 4 Royal Schools of Music Certification in Piano.
• My favourite Warhammer 40K faction is the T'au!

---

Thank you for visiting my profile! Feel free to reach out if you're interested in collaborating or want to chat.

kal-cal's Issues

Flux Suppression Simulations

Hi all, so I set up a script to convert model arrays to fits images. I had not even gotten to recovered flux arrays when I spotted the issue.

Kat7 "True" model data - 2 sources
Looks fine and probably why it did not look so bad in our results.

Meerkat "True" model data - 2 sources
One of the sources isn't even in the image, let alone the fact that one is not at the centre.

VLA-B "True" model data - 2 sources
Both sources are definitely not in the correct place.

This is only the true model image made from the skymodels. Obviously the gridding from the skymodels is not working, so instead of showing the recovered flux fits images and trying to fix this conversion problem, I think I will just follow what you said @landmanbester and create the array myself with no use of skymodels. That way I can ensure all sources are accounted for and I can fix the image size, plus ensure no overlapping sources. Am I mistaken with any of this?

**Edit: I've yet to put the skymodels over the image, but I don't know if that is possible with radio-padre and still trying to get X11 to port to my desktop for tigger.

Procedurally generate UVW data

The simplest route for me (for now) with the testing is to generate my own data that would come from a measurement set (using simms). Right now, I am trying to figure out how I could possibly create the UVW data with shape (n_row, 3). The code that fetches it from the table for data simulation is as follows:

...
    # Get convention
    if args.phase_convention == 'CASA':
        uvw = -(ms.UVW.data.astype(np.float64))
    elif args.phase_convention == 'CODEX':
        uvw = ms.UVW.data.astype(np.float64)
    else:
        raise ValueError("Unknown sign convention for phase")
...

where it is then used to generate model-visibilities using the africanus.dft.dask.im_to_vis. How would I go about trying to do this?

Terrifying package installing text

I've only tried this with the venv within kal-cal, but when I run:

pip3 install kal-cal

I get the following huge wall of text from trying to download require packages:

Processing /home/brian/Code/kal-cal
Requirement already satisfied: codex-africanus==0.2.10 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (0.2.10)
Requirement already satisfied: dask>=2021.1.0 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (2021.1.0)
Requirement already satisfied: dask-ms>=0.2.6 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (0.2.6)
Requirement already satisfied: llvmlite==0.35.0 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (0.35.0)
Requirement already satisfied: matplotlib>=3.3.4 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (3.3.4)
Requirement already satisfied: numba>=0.52.0 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (0.52.0)
Requirement already satisfied: numpy>=1.19.5 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (1.19.5)
Requirement already satisfied: astro-tigger>=1.5.0 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (1.5.0)
Requirement already satisfied: astro-tigger-lsm>=1.6.0 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (1.6.0)
Requirement already satisfied: astropy>=4.1 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (4.1)
Requirement already satisfied: pytest>=6.2.2 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (6.2.2)
Requirement already satisfied: pytest-forked>=1.3.0 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (1.3.0)
Requirement already satisfied: pytest-xdist>=2.2.0 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (2.2.0)
Requirement already satisfied: python-casacore>=3.3.1 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (3.3.1)
Requirement already satisfied: PyYAML>=5.3.1 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (5.3.1)
Requirement already satisfied: scipy>=1.5.4 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (1.5.4)
Requirement already satisfied: simms>=1.2.0 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (1.2.0)
Requirement already satisfied: Tigger>=0.1.1 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (0.1.1)
Requirement already satisfied: xarray>=0.16.2 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (0.16.2)
Requirement already satisfied: tqdm>=4.56.0 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (4.56.0)
Requirement already satisfied: packratt==0.1.2 in ./venv/lib/python3.6/site-packages (from kal-cal==0.0.1) (0.1.2)
Requirement already satisfied: decorator in ./venv/lib/python3.6/site-packages (from codex-africanus==0.2.10->kal-cal==0.0.1) (4.4.2)
Requirement already satisfied: appdirs>=1.4.3 in ./venv/lib/python3.6/site-packages (from codex-africanus==0.2.10->kal-cal==0.0.1) (1.4.4)
Requirement already satisfied: requests==2.*,>=2.23.0 in ./venv/lib/python3.6/site-packages (from packratt==0.1.2->kal-cal==0.0.1) (2.25.1)
Requirement already satisfied: jsonschema==3.*,>=3.2.0 in ./venv/lib/python3.6/site-packages (from packratt==0.1.2->kal-cal==0.0.1) (3.2.0)
Requirement already satisfied: importlib-metadata in ./venv/lib/python3.6/site-packages (from jsonschema==3.*,>=3.2.0->packratt==0.1.2->kal-cal==0.0.1) (3.4.0)
Requirement already satisfied: setuptools in ./venv/lib/python3.6/site-packages (from jsonschema==3.*,>=3.2.0->packratt==0.1.2->kal-cal==0.0.1) (53.0.0)
Requirement already satisfied: six>=1.11.0 in ./venv/lib/python3.6/site-packages (from jsonschema==3.*,>=3.2.0->packratt==0.1.2->kal-cal==0.0.1) (1.15.0)
Requirement already satisfied: attrs>=17.4.0 in ./venv/lib/python3.6/site-packages (from jsonschema==3.*,>=3.2.0->packratt==0.1.2->kal-cal==0.0.1) (20.3.0)
Requirement already satisfied: pyrsistent>=0.14.0 in ./venv/lib/python3.6/site-packages (from jsonschema==3.*,>=3.2.0->packratt==0.1.2->kal-cal==0.0.1) (0.17.3)
Requirement already satisfied: idna<3,>=2.5 in ./venv/lib/python3.6/site-packages (from requests==2.*,>=2.23.0->packratt==0.1.2->kal-cal==0.0.1) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.6/site-packages (from requests==2.*,>=2.23.0->packratt==0.1.2->kal-cal==0.0.1) (2020.12.5)
Requirement already satisfied: chardet<5,>=3.0.2 in ./venv/lib/python3.6/site-packages (from requests==2.*,>=2.23.0->packratt==0.1.2->kal-cal==0.0.1) (4.0.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./venv/lib/python3.6/site-packages (from requests==2.*,>=2.23.0->packratt==0.1.2->kal-cal==0.0.1) (1.26.3)
Requirement already satisfied: configparser in ./venv/lib/python3.6/site-packages (from astro-tigger>=1.5.0->kal-cal==0.0.1) (5.0.1)
Requirement already satisfied: astlib in ./venv/lib/python3.6/site-packages (from astro-tigger>=1.5.0->kal-cal==0.0.1) (0.11.4)
Requirement already satisfied: future in ./venv/lib/python3.6/site-packages (from astro-tigger-lsm>=1.6.0->kal-cal==0.0.1) (0.18.2)
Requirement already satisfied: astro-kittens in ./venv/lib/python3.6/site-packages (from astro-tigger-lsm>=1.6.0->kal-cal==0.0.1) (1.4.3)
Requirement already satisfied: toolz>=0.8.2 in ./venv/lib/python3.6/site-packages (from dask>=2021.1.0->kal-cal==0.0.1) (0.11.1)
Requirement already satisfied: python-dateutil>=2.1 in ./venv/lib/python3.6/site-packages (from matplotlib>=3.3.4->kal-cal==0.0.1) (2.8.1)
Requirement already satisfied: pillow>=6.2.0 in ./venv/lib/python3.6/site-packages (from matplotlib>=3.3.4->kal-cal==0.0.1) (8.1.0)
Requirement already satisfied: cycler>=0.10 in ./venv/lib/python3.6/site-packages (from matplotlib>=3.3.4->kal-cal==0.0.1) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in ./venv/lib/python3.6/site-packages (from matplotlib>=3.3.4->kal-cal==0.0.1) (2.4.7)
Requirement already satisfied: kiwisolver>=1.0.1 in ./venv/lib/python3.6/site-packages (from matplotlib>=3.3.4->kal-cal==0.0.1) (1.3.1)
Requirement already satisfied: iniconfig in ./venv/lib/python3.6/site-packages (from pytest>=6.2.2->kal-cal==0.0.1) (1.1.1)
Requirement already satisfied: toml in ./venv/lib/python3.6/site-packages (from pytest>=6.2.2->kal-cal==0.0.1) (0.10.2)
Requirement already satisfied: packaging in ./venv/lib/python3.6/site-packages (from pytest>=6.2.2->kal-cal==0.0.1) (20.9)
Requirement already satisfied: pluggy<1.0.0a1,>=0.12 in ./venv/lib/python3.6/site-packages (from pytest>=6.2.2->kal-cal==0.0.1) (0.13.1)
Requirement already satisfied: py>=1.8.2 in ./venv/lib/python3.6/site-packages (from pytest>=6.2.2->kal-cal==0.0.1) (1.10.0)
Requirement already satisfied: zipp>=0.5 in ./venv/lib/python3.6/site-packages (from importlib-metadata->jsonschema==3.*,>=3.2.0->packratt==0.1.2->kal-cal==0.0.1) (3.4.0)
Requirement already satisfied: typing-extensions>=3.6.4 in ./venv/lib/python3.6/site-packages (from importlib-metadata->jsonschema==3.*,>=3.2.0->packratt==0.1.2->kal-cal==0.0.1) (3.7.4.3)
Requirement already satisfied: execnet>=1.1 in ./venv/lib/python3.6/site-packages (from pytest-xdist>=2.2.0->kal-cal==0.0.1) (1.8.0)
Requirement already satisfied: apipkg>=1.4 in ./venv/lib/python3.6/site-packages (from execnet>=1.1->pytest-xdist>=2.2.0->kal-cal==0.0.1) (1.5)
Collecting argparse
  Using cached argparse-1.4.0-py2.py3-none-any.whl (23 kB)
Requirement already satisfied: pandas>=0.25 in ./venv/lib/python3.6/site-packages (from xarray>=0.16.2->kal-cal==0.0.1) (1.1.5)
Requirement already satisfied: pytz>=2017.2 in ./venv/lib/python3.6/site-packages (from pandas>=0.25->xarray>=0.16.2->kal-cal==0.0.1) (2020.5)
Building wheels for collected packages: kal-cal
  Building wheel for kal-cal (setup.py) ... done
  Created wheel for kal-cal: filename=kal_cal-0.0.1-py3-none-any.whl size=30288 sha256=b66059276c5496420acfd1a8d97ea2ceacc0669dd941b642c7142af1690bb3c0
  Stored in directory: /home/brian/.cache/pip/wheels/10/85/df/5061ed174f6ec3a87284f98200d4793f5559069ca2532796fe
Successfully built kal-cal
Installing collected packages: argparse, kal-cal
  Attempting uninstall: kal-cal
    Found existing installation: kal-cal 0.0.1
    Uninstalling kal-cal-0.0.1:
      Successfully uninstalled kal-cal-0.0.1
Successfully installed argparse-1.4.0 kal-cal-0.0.1

I realized this might be overkill for requirements as I ran pip3 freeze > requirements.txt for the requirements list. I'm not too sure how python handles package dependencies so I just listed all as required. Any advice on how to mitigate this so its something pretty like:

Installing collected packages: numpy, matplotlib, dask, kal-cal # and so on
  Attempting uninstall: kal-cal
    Found existing installation: kal-cal 0.0.1
    Uninstalling kal-cal-0.0.1:
      Successfully uninstalled kal-cal-0.0.1
Successfully installed argparse-1.4.0 kal-cal-0.0.1, numpy, matplotlib, dask # and so on