I'd say generally if incorporated in a lined stud wall that I'd be happy taking an effective length for buckling in the plane of the wall as the nog spacing. This would for typical stud wall heights found in residential type construction effectively force a column or studs to be typically critical for buckling in the out of plane direction.
If not in a lined wall or in a very short lined wall I'd tend to take the effective height as the full height between end restraints (typically top or bottom plates). In this case there is the liklihood that the column and the studs will buckle over the full height all together without a higher mode of buckling between nogs being developed.
In my own timber code this seems to be very much left to the judgement of the engineer.
Like any buckling, something has to be provided to take the restraint forces that develop to force the higher mode of buckling. While not specifically assessed in typical design, I'd say the linings play an important role in just making this work without too much extra justification.
