A formation of two F/A-18A (Hornet) aircraft were tracking from Townsville to Williamtown on airway Y124, maintaining Flight Level (FL) 350. A Boeing 767 (B767) tracking from Brisbane to Darwin on airway A464 was also maintaining FL350. The tracks of the two airways crossed at a position approximately 80 NM north-west of Taroom. At 1127, the crew of the B767 contacted the sector 5D radar controller, on climb to FL350. At 1135 (approximately 100 NM north of the crossing point) the Hornet formation, also maintaining FL350, made initial contact with sector 5D. Numerous exchanges occurred between the formation and the 5D controller with reference to the formation's track and confirmation of their next waypoint. The controller had a moderate workload due to other traffic operating in the sector at the time. At 1149, the radar controller recognised that the proximity of the Hornet formation and B767's position symbols might result in a loss of separation, and issued the crew of the B767 with an instruction to immediately descend to FL330. Recorded radar data indicated that the B767 and the Hornet formation were 20 NM apart at that time. Once the crew of the B767 had responded to the descent instruction the controller issued traffic information about the Hornet formation. The controller then made three attempts to contact the Hornet formation. The first two were to issue a climb instruction, however, no response was received. The controller then issued a traffic statement to the formation regarding the conflicting traffic, but again received no response. The crew of the B767 advised that they had received traffic alert and collision avoidance system (TCAS) indications and questioned whether the Hornet formation had responded to the control instructions to climb. Radar data showed that the B767 passed behind the Hornet formation with a minimum separation of 4.9 NM. At that point, the B767 was passing FL333 on descent. The required separation standard was either 5 NM lateral or 2,000 ft vertical separation. From the time the Hornet formation initially reported on frequency, there were at least 16 transmissions with reference to the formation's tracking and controller/aircrew confirmation of a track waypoint. During a subsequent interview, the controller advised that he was concerned that the formation was off-track, and continuing to diverge to the right, because this had the potential to bring the aircraft into conflict with traffic on a northbound route. This distraction may have led to a reduced scan rate on the radar and late recognition by the controller of the impending confliction with the B767.