Literary experts have sorted fiction in the story or novel form into two categories: literary and genre. In essence, "literary fiction" is what you learn in school, as "genre fiction" is what you buy and enjoy during your off time. Shakespeare would be literary, while Harry Potter would be genre. My take on this: there are better terms that could be used to define both categories. I hold a grasp on the fact that what we learn in high school and college and what wins the Nobel Prize in Literature versus what we read for pleasure and brings in the money or media recognition is incredibly different. I know that two terms can be used to describe these, but I don't see literary and genre fiction being the two.
Literary and genre both make up each and every element in the creation of fiction. Literature is the art of written work. There is no specification as to what the written work be about, but it needs to be of strong enough quality that you could paint a picture into the reader's mind. Of course, some literature is of better quality than other literature. It all comes down to the elements that are included in the story.
A genre is a category. Categories in fiction can fall in categories such as horror, science fiction, mystery, romance, young adult, children's, historical, among plenty of others I did not mention. There are more general categories such as drama, in which many of the literary works would go. Though there are plenty of literary works that can be categorized. Nobody goes away without a label, it's just like high school, where cliques are basically labels that you are given based off of your train of thought and the path you wish to pursue in your little society.
There are plenty of authors that are deemed "literary fiction" that could easily be placed in categories. We learn about Edgar Allan Poe in school, even going as far as having a unit named for him. He happens to be a well-known pioneer in horror fiction, a lesser known pioneer in science fiction, and with "The Murders In The Rue Morgue," the father of mystery. He is, without a doubt, one of the most important figures in literature, but his writing can be categorized. Stephen King writes in the same genre of horror, though he does stray to other categories such as fantasy. There is only one difference between the way Edgar Allan Poe and Stephen King write and that difference is time. Perhaps, by 2050, Stephen King's work will be taught in English classes. It is indeed literature, just literature that's more current.
I believe a more appropriate term for what is deemed as "genre fiction" would be "pop fiction." The inspiration for this term has much to do with how we consider the music we listen to on the radio is pop or contemporary. However, "pop fiction" would be difficult to categorize, unless you want to use the Barnes & Noble arrangement as an example. Plenty of books are categorized as "Fiction and Literature," while the pieces that are slightly more dedicated to their genre are found in their appropriate section. However, in a section like science fiction and fantasy, what you will find is everything that fits the category. You won't only find "pop-based" science fiction and fantasy that is deemed "genre fiction," but instead anything that fits the category. You will find Frank Herbert and Jim Butcher, but you will also find Ray Bradbury, Jules Verne, and H.G. Wells. You'll even find H.P. Lovecraft, and he has been declared by plenty of lists the greatest horror fiction writer of all time.
I really believe there should not be such a thing as "genre fiction," because everything could be categorized in its own genre. Sure, it's easier, but once we dissect what is deemed as "literary fiction," then two categories no longer exist. I could start with how much of ancient world literature was adventure or mythological, such as Homer's epics or the Oedipus pieces by Sophocles. Robinson Crusoe, the first ever novel written in prose format, has to be an adventure. Plenty of the pieces deemed as "literary" are simply dramas, like Jane Eyre. Stephen Crane wrote military or Western fiction, Toni Morrison writes race fiction, and Shakespeare writes plays, primarily comedy or tragedy depending on the ending, but can also be deemed as romances, because the story is typically one subject falling in love with another. He holds no concern in the subject of prose.
I would argue that just because something sits in a specific category doesn't mean that it cannot be deemed as literature. Everything that's written is literature. Even if I dislike the work, the work is still literature. It's not very good literature, but it's literature nonetheless. Of course, there is a difference between what wins the Nobel Prize in Literature and what becomes a bestseller, but everything can easily be categorized with the necessary time and tools.