Markus Pobitzer

Masterthesis at IBM research

We propose a method to augment small, annotated instance segmentation datasets to effectively obtain a sizeable, annotated dataset. We generate new images using a diffusion-based inpainting model to fill out the masked area with a desired object class by guiding the diffusion through the object outline, preserving the provided mask annotations. Work submitted to WACV 2024; IBM Research started the process of filing for a patent.

On-body touch input in VR using wrist-watch touch detection

My Bachelor thesis at the SIPLAB

Modern interaction methods in virtual reality (VR) are based on controllers or in-air gestures. This overlooks smartwatches and wearables strapped on the wrist as a complement to inair gestures. In this thesis, we quantify the effect of haptic on-body input in VR on accuracy and user experience for menu item selection.

SFSCON 2023 - Image Generation with Diffusion Models

Recent machine learning developments saw a breakthrough in generating images. So-called Diffusion Models can create photo-realistic images from noise. With the help of an input text (prompt) we can guide the generation and produce matching images.

This technology opened new doors for creating digital art, modifying existing images, and creating stunning visual experiences. In the talk, we will find out how these algorithms work, introduce Stable Diffusion (a concrete implementation), and explore what its use cases are.

Stable Diffusion RePaint

Bringing RePaints capbilities to Stable Diffusion.

Contributing to Hugging Faces Diffusers community pipeline.

Example usage can be found here.

RGB-D NERF

Virtual reality (VR) and augmented reality (AR) immerse the user in a new digital world. However, representing realworld scenes and objects digitally is very challenging. Realistic lighting and high details are hard to model. An approach that solves some of the mentioned shortcomings was introduced with Representing Scenes as Neural Radiance Fields for View Synthesis (NeRF). NeRF can produce photorealistic novel views but needs many RGB input images to train. In this work, we explore how NeRF can be extended with synthetic depth information to reduce the needed number of input images.

Sentiment analysis of tweets (Competition)

1st place in the Twitter sentiment analysis competition hosted by the Computational Intelligence Lab at ETH Zürich. For the full competition and the placing view Kaggle.

In the paper, we explore different deep learning models, among them state-of-the-art Transformers and their combinations in different ensemble methods to achieve the winning score.

Real-time Neural Human Rendering in VR

We propose a point-based neural rendering pipeline developed for real time applications in the virtual reality (VR). We use point clouds as our representation of the human models and propose an adaptive shader to render the point clouds in Unity. Through Unitys neural network inference library we pass the rendered images through a neural network to enhance the output

Offline Python SLAM using COLMAP

The goal of this project is to combine COLMAP with ideas of a SLAM pipeline to obtain a fast, robust, and scalable SLAM system. We demonstrate that we partially achieved our goals by utilizing PyCOLMAP and ideas from ORB-SLAM.

”Seeing” Temperature - Integration of IR camera into the HoloLens 2

We provide a system that modifies the physical build of a Microsoft HoloLens 2 such that it holds Hardware components capable of measuring temperature. Furthermore, it uses a TCP connection to send data from a Raspberry Pi to the HoloLens, processes this data, and provides different modes for the user to visualize the thermal data.