Spent some time last night and got the tracking working again after all the changes. For now, will do tracking and streaming in the same time to be able to see what camera sees. Later, can disable the video streaming and keep only tracking to be able to track at very high-speed.
This is done using an Atmega16 chip with a OV7670 (FIFO model). Colour segmentation is applied on YUV signal rather than RGB as it seems less affected by light variation. So far works pretty fine and is cleaner then my first prototype, however, now I am pulling the data (host start the tracking by setting the resolution, image orientation, white balance, etc then trigger tracking for every frame and CTS module will reply back with the information, once frame processed/sent will wait for CTS to start another operation/tracking) while before was in a continuous loop. This mode can be achieved by closing the jumper on the CTS board.