Stability is more more important in any vehicle. Especially for vehicles which fly in air . We need some extra focus . For any structure to be stable ,they must have equal and distributed reaction forces. Center of Pressure and Center of Mass decides weather the rocket move in upward or will it move in random direction

Center of Mass 

Center of mass is the mass from the which the mass is distributed uniformly throughout material(rocket here) . The rocket in space rotates about the center of point . The axis of rotation goes through center of rotation .

Center of Pressure 

Center of pressure is roughly considered as the center point of surface area of rocket. 

To make Rocket aerodynamically stable , The Center of Mass always must be above the center of pressure ( when seeing from Nozzle side) . This the stability condition for rocket. 

And This is the reason why rockets use Fins 
These fins make the COP to come down and to increase the stability of rocket.

It is the same reason that why use a long stick of light weight is used for cracker rockets. This stick makes the COP to come down than the COM.