At 0758 EST, VH-OUD contacted Brisbane Approach 28km north of Brisbane at 3,000 feet requesting an airways clearance. Approach answered that a clearance was not available and that the aircraft was required to remain outside controlled airspace. Approach also told the aircraft to contact the Redcliffe common traffic advisory frequency (CTAF) and to call Approach again at 0805 EST. Another aircraft, VH-HEI, was on descent inbound to Brisbane and had been assigned 3000 feet. As the progress of this aircraft was being monitored on radar, the Approach Controller realised that VH-OUD had penetrated the control area. Attempts to contact VH-OUD were unsuccessful so VH-HEI was vectored clear of VH-OUD. The separation standard of 5 miles was infringed during this sequence. The pilot of VH-OUD contacted Approach again about 5 miles north of Brisbane Airport and was processed for landing. The pilot later said that he heard the instruction to remain outside controlled airspace. A short time later, however, he had been told to squawk ident on the aircraft transponder and was informed that VH-OUD had been radar identified. He interpreted this to mean that he could continue into controlled airspace at 3,000 feet.