It can be very difficult, in general. An example is the book of Jonah. Taken literally, he spends three days and nights inside the belly of a fish or whale. Other readers believe that the book is not meant to be read historically, but is a kind of fable that carries a moral lesson. How are we to determine which is the case?
- Read the book and find out what it says about itself.
- Compare it to other writings, prophetic or historical or literary.
- See how it is used elsewhere in the Bible (such as Luke 11:29-32).
- Examine the language and history - When did Jonah live (2 Kings 14:25)? Did he write the book, or is it only attributed to him? Is the language characteristic of that era? Is Nineveh as large as Jonah says? Did Assyrian kings behave in the way suggested? Can a person live inside a whale?
- Find out what experts have written about the book. Consider their evidence and arguments.
- Think about: What would the implications be if this were historical? What if it were not historical? Either way, how does that fit with any other aspect of my faith?
Additionally, while some passages are clear, there are many that have multiple meanings. I decided to find an example by opening my nearest Bible (NJB) at random. It came open at the first page of Hosea, where we read (1:2)
Yahweh said to Hosea, "Go, marry a whore, and get children with a whore; for the country itself has become nothing but a whore by abandoning Yahweh."
The family life of Hosea (literal, historical) has an additional layer of meaning (allegorical, figurative) explained in the text: it mirrors the relationship between God and his faithless people. Equally, Hosea was speaking to Israel at a particular time, but his theme is applicable to us in general, as individuals and as a church. This more general interpretation may also be justified in terms of the Bible text as a whole: Jesus quotes Hosea 6:6 in Matthew 9:13 and 12:7, clearly validating the wider application of that passage.
This kind of thing is very common; it does not necessarily make sense to make a clean demarcation between "historical" and "figurative" passages, when there are so many instances of the Bible saying "X happened, and this is what X means". The circumstances of the Fall, the Exodus, the Exile, etc., are continually being referred to in newer writings, and used to draw moral or theological lessons.
The really difficult part comes with passages like Joshua 10:12-15. This is the part where "the sun stood still in the middle of the sky and delayed its setting for almost a whole day" (10:13). The literal meaning is, of course, that the sun did just that (and the moon too). It is presented in the text as genuinely happening, as a historical event. We can draw wider lessons, like - the sun is subject to God, God protects his people, and so on. It's not hard to believe that this passage is figuratively true in that regard, but many people have difficulty believing in its literal truth. So this question goes right to the heart of how one reads the Bible, and whether it is possible to believe in the wider message without believing in its historical grounding. In short, the determining of literary types implies also being able to settle many other controversies of Biblical interpretation.