Personally, I knew from the get-go that fantasy was going to be my genre. This, of course, is my favorite genre to read, closely followed by science fiction, which is my other main area of writing.
Which leads me to my first piece of advice: choose a genre you love to read. This will be more natural to you. You love the genre already, and you know the ins and outs of it, which will also help you when it comes to publishing and marketing, because you'll be aware of the market.
It's also important to note that the idea comes first. If you've got a good story idea, it should naturally fall into a genre of some kind. Most of my ideas are in the fantasy realm, with some moving into the grey area of speculative fiction, and others further into science fiction. It's rare that I sit down and actually say, "I want to write a YA fantasy novel." Rather, the idea appears, it fits a genre, and I write it as such.
For me, the most spectacular example of this was when I tried to write realistic fiction, back in middle school. I finished the novel, all right, but it was really, really bad. Personally, I need the excitement and magic and the more metaphorical nature of fantasy and sci-fi. You strip away all of the speculation, and suddenly everything falls flat, and I've got absolutely nothing to say. It makes sense; I write fantasy because that's actually sort of how I see the world. Maybe that's how it'll work for you too: maybe you should write in a genre that reflects the way you think.
With those tips in mind, here's a list of different genres and sub-genres, with examples (some of which I have read, some of which I haven't). This should give you a better idea of what's up!! As you'll see, there's a lot of overlap between genres. Also remember, you can mix and match these terms if it better describes your book, although you shouldn't use more than a couple when you're pitching to agents and editors.
A genre with fantastical elements that don't exist in the real world, such as magic or supernatural forces.
Epic/High Fantasy: The most familiar kind of fantasy, with a heroic journey, famous archetypes, and most likely some kind of quest and/or a fight between good and evil. (The Lord of the Rings, The Chronicles of Narnia)
Contemporary Fantasy: A combination of contemporary fiction and fantasy, i.e. a fantastical story that takes place in the modern day and often in a real, rather than imaginary, world. (Harry Potter, The Borrowers)
Urban Fantasy: A contemporary fantasy that takes place in an urban setting. (Unearthly, House of Night)
Dark or Gothic Fantasy: A combination of horror and fantasy that often has a gloomy or moody feel. (The Vampire Chronicles, Shadowland)
Paranormal Fiction: Focuses on paranormal creatures such as werewolves, vampires, and ghosts, but without the fear factor of the horror genre. (Twilight, Shiver)
Magical Realism: A otherwise realistic story with strange hints and dabs of fantasy; originates in Latin American culture. (Bone Gap, A Very Old Man With Enormous Wings)
Fairytale, Fable, or Mythic Retelling: Retells a cultural-specific and often long-historied tale; may fall into a genre other than fantasy. (Percy Jackson, Cinder)
A genre where the story is told in order to frighten the reader, often with a fantastical element.
Ghost Story: Revolves around ghosts, spirits, and occasionally, revenants. (The Shining, The Monkey's Paw)
Monster Story: Revolves around some kind of fantastical monster. (Frankenstein, Dracula)
Survivalist Fiction: Focuses on humans trying to survive in a world overrun by a group of fantastical monsters, such as zombies. (The End Game, The Hunt)
Occult Horror: Revolves around demons, witchcraft, and Satanists. (Macbeth, Speak Daggers to Her)
Slasher Fiction: The most realistic form of horror; very much about the blood, gore, and violence and often features a psychopathic killer. (Friday the 13th, Texas Chainsaw Massacre)
A genre where the protagonist journeys to specialized locations and often ends up in a desperate situation.
Military Fiction: Centers around a war, fictional or real. (The Things They Carried, A Farewell to Arms)
Spy Fiction: Centers around a secret agent or military member sent on an espionage mission. (James Bond, Gallagher Girl)
Western: Focuses on cowboys, Indians, outlaws, and all that sort of gun-slinging action. (Jesse James, The Savage Guns)
A genre that takes place in a historical setting, often centering around real events and generally labelled with the time period it falls into.
Alternate History: A combination of speculative fiction and historical fiction where the author tries to answer a "What If?" about an important point in history. (Aristopia, If It Had Happened Otherwise)
Period Piece: A form of historical fiction where the setting is merely a backdrop; the story runs on its own with no focus on a major historical event. (Cinderella Man, The Western Canon)
A genre that takes place in the realistic, modern-day world.
Literary Fiction: High-brow and focused on deep, philosophical concepts; a bit of a sexist term, as it's usually applied to contemporary fiction written by men.
Women's Fiction: Focuses on the experience of one or more women; a bit of a sexist term, as it is usually applied to contemporary fiction written by women and is treated as more low-brow.
A genre with elements that don't exist in the real world, but which have scientific explanations.
Hard Sci-Fi: Has very detailed, well-researched, and realistic science that drives the storyline. (Jurassic Park, Prey)
Soft Sci-Fi: Focuses more on political and social concepts and less on the science. (1984, Brave New World)
Apocalyptic/Post-Apocalyptic Fiction: Takes place during or after an apocalyptic event and examines the causes and effects of such an occurrence. (Gone, Life as We Knew It)
Dystopian Fiction: Examines societies and worlds which are, essentially, "nightmare worlds;" is often post-apocalyptic in nature. (The Hunger Games, Divergent)
Space Opera: Takes place almost entirely in space and revolves around space travel. (Star Wars, Star Trek)
Cyberpunk: Features humans who have been enhanced by cybernetic technology. (Neuromancer, The Blade Runner)
Steampunk: A combination of historical fiction and sci-fi where some important part of history is turned speculative; often focuses on the use of steam engine technology. (Leviathan, The Time Machine)
Superhero Fiction: Revolves around characters with superhuman abilities acquired through science and/or magic. (The Adventures of Superman, X-Men)
Speculative Fiction: A combination of sci-fi and fantasy, and sometimes horror, that doesn't quite fall into any of these genres. (The Yellow Wallpaper, The Outsider)
A genre between action-adventure and horror which incites both fear and excitement through a more realistic story premise.
Disaster Thriller: Features some kind of mass peril the protagonist must save others from, such as a terrorist attack or natural disaster. (State of Fear, Ashfall)
Medical Thriller: Focuses on some sort of medical disaster, often with a limited amount of time to resolve the issue. (Contagion, Coma)
Psychological Thriller: Explores the mind of the protagonist; is often written in a way that confuses and misleads the audience. (Black Swan, Inception)
Crime Thriller: A combination of the crime fiction and horror/thriller genres. (Vertigo, Jack Reacher)
A genre between action-adventure and thriller that focuses specifically on crime.
Detective Story: Tells the story of a detective solving a crime: the who, why, and sometimes how. (Death on the Nile, The Drowning Pool)
Murder Mystery: A detective story focusing specifically on a homicide case. (I Hunt Killers, Dark Places)
Courtroom Drama: Usually television-oriented, this focuses on crime and the law. (Law & Order, Boston Legal)
Gangster Fiction: Focuses on gangs and other criminal organizations. (Cold Fury, The Godfather)
A genre, often used in combination with other genres, where the focus of the story is on one or more romantic relationships.
Contemporary Romance: A combination of contemporary fiction and romance. (The Princess Diaries, All-American Girl)
Historical Romance: A combination of historical fiction and romance. (Pride and Prejudice, The Fires of Winter)
Fantasy Romance: A combination of fantasy and romance. (Beautiful Creatures, Hush Hush)
Sci-Fi Romance: A combination of science fiction and romance. (The Host, Perdition)
Erotica: A romance that focuses on the sexual aspect of the relationship. (50 Shades of Grey)
Diverse: Features characters of various real-world identities--sexualities, races, disabilities, religions, etc.
Own Voices: Features a main character with a marginalized identity; written by an author with the same identity.
What's your favorite genre to read? What category and genre do you write?
Images via clarklib.org, sciencefiction.com, wikihow.com, radiotimes.com, theguardian.com, article.wm.com, and mirror.co.uk.