Last night managed to spend some time to redo my panning code and got it moving much better. Still work to be done but is not too bad.
Then got a CF7670C-V1 module work with this setup, also tracking at high-speed using the same piece of code, by doing a short-circuit between pin5 and pin6 on the module. This way will behave like a V2 one, however here is no HREF signal going out, not used in my application.
Next will be trying to get it running with LPC1114 chip, cause this microcontroller is very cheap and can cut down the BOM quite a lot if I will decide to make a commercial version for education purposes. Also the extra RAM can help in tracking multiple blobs of same or different color.