I want to use a check mark image in the formula field. I got confused about where to store the image: in static resources or in documents. Please explain to me the use cases of both.
Documents is meant as a reference for the typical end Users to use when necessary. This does not transfer between orgs.
Note that if you want to make say an image used in a HTML template available to the unauthenticated recipient of the email then you must use a Document as described in How do I reference a public image stored in salesforce.com in my HTML?.
Short answer, if you are using it for a check mark image that will be used in a field, this needs to go between orgs and so I would use a static resource if possible.
If you are willing to use the default Salesforce checkmark image, then having this in a formula field having return type as 'text' should work.