The pilot was engaged on his second charter flight for the day from Darwin. A clearance for an immediate takeoff was given but the pilot delayed on the taxiway until prompted by the tower controller. The departure clearance was via Lee Point to the north but the aircraft was observed tracking to the west. As there were three other aircraft in the circuit, the pilot was again instructed to track via Lee Point which he acknowledged. The aircraft continued to track west and was instructed to turn onto a heading of 360 degrees to avoid circuit traffic. A short time later the aircraft made a 180 degree turn and advised the tower he was returning to Darwin. A traffic alert was given by the tower controller and the other aircraft in the circuit altered course to prevent a breakdown in separation. Being unfamiliar with the area, the pilot stated that he had confused East Point (west of Darwin) with Lee Point. The previous evening, the pilot had worked at a part-time position until 0100. He then worked from 0600 until 1100 on the day of this flight before commencing his rostered flying duties.