PROJECT 1: RGB-D Fusion for Segmentation on Indoor data for Autonomous Wheelchair

Research Project - Worked on image+depth fusion techniques with transformer networks and with traditional YOLOV7 for segmentation on indoor dataset used for autonomous wheelchair scenario.

Problem - Automate the motorized wheelchair by creating an understanding of the surrounding scene

Solution - Using various segmentation methods, analyze the pros and cons of the outputs based on the model architectures, and improve the performance on the collected dataset.

• With the goal of contributing to the automation of a motorized wheelchair, this research project focuses on deep learning approaches and data fusion techniques on indoor data. The objective of the research project is to perform segmentation on indoor dataset collected manually from a camera at Carleton University.


• I mainly explore and compare two segmentation methods, traditional neural networks Yolov7 and a multimodal encoder-decoder transformers TokenFusion, in terms of performance and outputs.


• Yolov7 : solely depends on image inputs with traditional CNNs for segmentation tasks. Originally trained on a much larger dataset (MS Coco dataset) of only images. Yolo originally performs object detection, but version 7 can also perform segmentation.


• TokenFusion: transformer method depends on multimodal data (Image and depth), fusing images with corresponding depths in the encoder-decoder models. Originally trained on a smaller dataset (Nyudv dataset). Can perform both segmentation and object detection


• The results demonstrate that the inclusion of depth images are highly beneficial for segmenting the interior structure of the indoor environment, and the performance based on training, time and accuracy are also compared between the two models. Both the methods are quite different in terms of input modalities and backbone networks, and the advantages and disadvantages of both methods are discussed.


• Questions I asked myself while using the ML model:
• All these models are shown to perform really well on the dataset they were trained on, but does it actually perform well on ANY indoor dataset? Let's explore!
• If it doesn't, what are the possible reasons? What should be fine-tuned?
• What information is it failing to understand or percieve properly?
• Is this a generalization problem?
• Is this a data preprocessing problem?

*If you're interested to discuss more or view the code, please Email Me . Due to privacy constraints, you will need extra permission from the Professor to view the code.

PROJECT 2 : GANs (Generative Adversarial Networks) on Limited Data

Problem : Shortage of Image data with labels

Solution : Synthetically generate image data with Generative AI Models

• Used pre-trained StyleGAN2, GP-GAN and SIN-GAN trained on Breast Cancer images.


• It is important to see how the model performs inititally without fine-tuning. To no surprise, the model performed horrible initially. But this gives an insight to where the problem might be and what needs to be changed.


• Fine-tuned learning rates and decay values


• Changed the regularizor loss to LeCam Loss to improve the generalization of the images.


• Synthetically generated better images based on different batch sizes


• If more GPU power was available, it is possible to attain further detailing of the generated images.


• Questions that still need to be answered:
  • How do you defined good results here? I know the loss and FID metrics provide an insight, but even for us humans, it is not clear how to measure goodness of abstract images. If you have any insights, please Email Me .

View on Github