Tolleson’s life in football began at Anniston High School where he played first for Freeman (Bulldog) Johnson and then Bill Rutherford. At Anniston, he lettered three years as a fullback and linebacker and was a permanent team captain during his senior season in 1965. 

While at Anniston, Tolleson was a two-time All-Calhoun County player and a two-time Class 4A All-State selection. He lettered two years in track where he won county, conference and regional shot put titles. 

Tolleson received a football scholarship to Tampa and lettered two years as a running back. He transferred to Delta State where he played linebacker and lettered two more years. 

He remained at Delta State as a student assistant for the 1970-71 school year, beginning his 37- year career coaching football. 

Defensive coordinator at Lineville High School in the 1971 football season was his next job. After one year with the Aggies, he returned to school to earn a graduate degree at Livingston University, now West Alabama, and coached there as a graduate assistant. He then spent three more seasons with high school programs, one at Guntersville and two at Minor, as defensive coordinator. 

Livingston made him its defensive coordinator and strength and conditioning coordinator in 1976 and he remained with the Tigers for three seasons. After serving as assistant head coach and defensive coordinator at Louisiana Tech in 1979, Tolleson moved to the University of Arkansas where he coached linebackers and defensive line for four seasons. 

From Arkansas, Tolleson coached the defensive line for the Birmingham Stallions of the USFL. When the USFL disbanded, he entered private business for three years. Tolleson returned to coaching in 1989 as inside linebackers coach at Stanford. Positions at Texas-El Paso, Southern Mississippi and LSU followed over the next nine years. 

His final 13 years of coaching came at the University of Texas. From 1998 through 2011, Tolleson coached defensive tackles and was special teams coordinator. His Longhorns were national champions in 2006. In 2009, Tolleson was named national special teams coordinator of the year.