Stone Mountain is the Atlanta suburb where I was raised, and is named for its chief landmark. Which is, as the name would suggest, a giant granite outcropping. Imagine a smaller, grayer version of Ayers Rock, located in the Deep South, with the images of three Confederate leaders carved into the side.
My enthusiasm soon turned to annoyance, though, once the characters actually arrived on the scene. An FBI snuck up on a cabin and entered, only to discover that their quarry had already left.
Something tells me that no one on the show checked to see what Stone Mountain actually is, beyond the name. Because Stone Mountain isn't an inhabited mountain; it's the centerpiece of a state park in metro Atlanta. Nobody lives there. There aren't any cabins to hide in, and even if there are some hiding up there, you couldn't utilize them (e.g., drive up the mountain) without tipping off the park rangers.
On the whole, this series may have taken its share of small liberties with the Atlanta area, but it definitely captures the city a lot better than The Walking Dead did.