When the aircraft left the control area on its previous flight, it was noted by ATC staff that the transponder was not operational. The operator was informed that the equipment was unserviceable but that the aircraft would be allowed into the control area on its return flight. However, when the aircraft reported outside the control area (OCTA) for a clearance it was denied on the basis that the transponder was unserviceable. This left the aircraft crew with the option of proceeding OCTA or landing somewhere else. The crew continued OCTA but inadvertently infringed the 3,000 step north of Cairns due to unfamiliarity with VFR procedures in the area and difficulty with the radio navigation chart presentation. As a result of this investigation, ATS has modified their approach to this type of situation. Aircraft crews will no longer be forced OCTA under similar circumstances. One of the problems inherent in the present radar system in Cairns is that only transponder returns can be seen beyond 50nm to the north-west due to terrain shielding of the primary radar. ATC's first option should have been to reroute the aircraft to track in from the north to give primary radar coverage earlier. A second option was to bring the aircraft in on a procedural clearance.