Judge sues Channel 5 for libel, setting stage for local test of uncommon legal maneuver