The rule of thumb is: can it read common GIS data formats in their native form? Or operate as a plugin for an application which is normally considered a GIS? (ArcInfo, ArcView, ERMapper, PCI, GRASS, MapInfo, Spring, SmallWorld, Manifold, TnTMips, etc.)
If not, it probably belongs in Graphics/3D/Rendering_and Modelling/Terrain.
For example ArcView 3D Analyst and Genesis II are appropriate, World Construction Set is not, (though EcoModeller for WCS is).
This category is for software which specializes in "non traditional" visualization of GIS data, which usually means 3D. In this context "traditional" visualization is cartography (maps) and 2d topographic display (lines and polygons) Which all GIS software can do.