Time travel actually is a scientific concept that doesn't violate general relativity. And since time is relative, time travel into the future is actually pretty simple - just move fast enough so that time around you dilates and thus moves slower in relation to your surroundings.
Time travel into the past is trickier, but doesn't violate GR either. The catch is that the most scientifically viable time travel methods (such as wormholes) require the creation of the method, which can then be used as a point of origin. In other words: you can travel into the past - but no further than the point that saw the "time machine" first being activated.
Which also explains why we have no time travelers walking in our midst today...