"Residual" method involves putting a relay element in the neutral (common) leg of three individual phase CTs in a wye connection. Any residual current flowing in the neutral of the wye represents ground (and/or) neutral current.
Zero sequence is also known as flux summation and involves putting a window-type CT around all three phase conductors. Normally, the flux of the three phase conductors should sum to zero so there will be no current in the CT secondary. If there is a ground fault, phase currents are not balanced, flux is not zero and there is secondary current in the CT proportional to the primary ground current.
Zero-sequence could also mean a calculated zero-sequence quantity from the phase currents. Schweitzer distinguishes these two by using Neutral Ground for current in the CT neutral (IN channel) and Residual Ground for a calculated 3I0 current.
You can obtain a much lower setting with a sensitive ground fault protection (zero sequence ground fault protection) than with residual ground fault protection.
A setting of 1%-2% Full Load Current is obtainable with sensitive ground fault protection while a setting of lower than 10% Full Load Current (residual ground fault protection) normally result in nuisance trips.
For additional info see also the NPAG, chapter 9, "Overcurrent protection for phase and earth faults"
