Projects (With Code)

Synthesis of Depth images from RGB images

Practical Project, Chair of Computer Graphics and Visualization, TUM, Munich

Depth images are a rich source information about their subjects and have been found to be particularly useful for tasks such as 3D Reconstruction. This work aims to learn a supervised pixel to pixel mapping from an RGB image to its corresponding depth image. The architecture in this work is based on the paper by Li, Jun, Reinhard Klein, and Angela Yao. “Learning fine-scaled depth maps from single RGB images.” arXiv preprint (2016)

For more details, see the code and the poster in the links below.

Semantic Segmentation of human depth images

Inter Disciplinary Project, Chair of Computer Aided Medical Procedures, TUM, Munich

This project is the consequence of an internal course requirement during my masters. The goal of this project, on its own, is to segment human body parts from depth images. Results from several different U-Net based models are compared in this work, with particular focus on high speed and accuracy.

Provided in this project are functions to obtain and preprocess the training data, train the segmentation graph, and freezes the graph to a protobuf file for later inference in C++.

Further development of this work (not included in this repository) focuses on combining the segmented depth maps using KinectFusion to provide a segmented 3D point cloud/mesh. This segmented 3D model of the human body was used in collaboration with some other techniques to provide a view into a patient’s body.

A detailed report on this work can be found in the link below.

Reimplementation of Importance Weighted Autoencoders

Practical Project, Chair of Robotics and Embedded Systems, TUM, Munich

Importance Weighted Autoencoders are a variant of Variational Autoencoders with a more powerful posterior. The Variational Autoencoder is a generative modelling technique that assumes an underlying latent structure to the data x. To successfully generate samples from the same distribution x was generated from, one must have a model of the data that maximizes p(x). However p(x) is the integral over p(x|z)p(z), which is intractable. Therefore, we instead maximize the lower bound by assuming a variational posterior (since we cannot tractably compute the true one or sample from it.)

Unfortunately, VAE assumes the variational posterior form to be factorial which restricts the capacity of the model to distributions with factorial true posterior. IWAE overcomes this assumption by sampling multiple times from the approximate posterior to get a tighter lower bound on p(x). The variational posterior provably approaches the true posterior in the limit value of infinite samples.

This work re-implements IWAE and VAE. The results and comparisons between these two models are laid out in a poster that can be downloaded by following the link below.

Iterative Closest Point Analysis

Course Project, Chair of Computer Vision, TUM, Munich

Iterative closest Point (ICP) is an algorithm employed to minimize the difference between two point clouds given an initial estimate of the relative pose. It is often used to reconstruct 2D or 3D surfaces from different scans, to localize robots and achieve optimal path planning and to register medical scans. ICP has several steps and each step may be implemented in various ways which give rise to a multitude of ICP variants. This project implements and analyzes several variants of ICP, comparing them on the basis of execution speed and quality of the result.

Dynamics with Inverse Autoregressive Flows

Practical Project, Chair of Robotics and Embedded Systems, TUM, Munich

Through this work, we propose the incorporation of Inverse Autoregressive Flows for determining the state space (latents) in a dynamical system model. This reduces the number of samples that need to be obtained in order to approximate the posterior distribution (and thus the underlying states/latents for a set of observations and controls) from one per time step to one per sequence of observations. Our experiments with pendulum-v01, an environment from openai gym confirmed that the accuracy with which the observations are generated are close to the state of the art for sequence models.