Transcranial magnetic stimulation (TMS) is a non-invasive brain stimulation technique whose effects on neural activity can be uncertain. Within the visual cortex phosphenes are a useful marker of TMS: they indicate the induction of neural activation that propagates and creates a conscious percept. In this work we systematically map out locations in the visual cortex where stimulation can trigger phosphenes. We relate this to the retinotopic organisation and the location of object- and motion-selective areas, identified by functional Magnetic Resonance Imaging (fMRI) measurements. We find that TMS can reliably induce phosphenes in early (V1, V2d and V2v) and dorsal (V3d and V3a) visual areas close to the interhemispheric cleft. However, phosphenes are less likely in more lateral locations (hMT+/V5 and LOC). This suggests that early and dorsal visual areas are particularly amenable to TMS and that brain stimulation can be used to probe the functional role of these areas.