Soft pastels, the most commonly used pastel, are made with a combination of white chalk, pigment and gum arabic, which gives it a drier matte finish. Soft pastels do not adhere to the surface, allowing it to be brushed off.
Oil pastels on the other hand, are similar to oil paints but don't dry out or harden completely. This enables you to add more layers of colors to blend them together. Also, try blending them with your finger or an artist's tool to move the oil pastels around the drawing. Using your finger is the recommended method for giving a nice finish while being hands on and having fun with the process.