In 1955, Richards was hired by the Baltimore Orioles, where he served as both field manager and general manager through 1958, becoming the first man since John McGraw to hold both positions simultaneously. As general manager, he was involved in a 17-player trade with the New York Yankees that remains the biggest trade in baseball history. Richards concentrated on signing good defensive players (such as Brooks Robinson) and hard-throwing young pitchers (such as Steve Barber, Milt Pappas and Chuck Estrada). After Lee MacPhail was hired as the general manager in 1959, Richards served strictly as the Orioles' field manager. The Orioles finally blossomed in 1960 with a second-place finish after five disappointing seasons. The Orioles’ second-place finish was Richards’ best as a manager. Both the Associated Press and United Press International named him the American League Manager of the Year.

This Reach official American League baseball was signed in black by Richards and includes a Great Moments COA. This baseball appears to have been shellacked.