Ilaiyaraaja was born into a poor rural family in Pannaipuram, Theni district, Tamil Nadu, India. He was the third son of Ramaswamy and Chinnathayammal. Growing up in a farming area, Ilaiyaraaja was exposed to Tamil folk music, such as the songs sung by farmers working in the fields. His formative contact with music-making and performance came at the age of 14, when he joined a travelling musical troupe headed by his elder step-brother, Pavalar Varadarajan, who was a propaganda musician for the Communist Party of India. He journeyed through numerous villages, towns and cities in South India with his brothers for about ten years as one of the musical Pavalar Brothers. He first tried his hand at composing music during this period: he set to music an elegy written by the Tamil poet laureate Kannadasan for Jawaharlal Nehru, India's first prime minister.

Arriving in Madras in 1968, Ilaiyaraaja enrolled under the tutelage of Dhanraj, a music teacher, as it became apparent that formal knowledge in music such as musical notation was vital for a professional music career. He was introduced to Western classical music during his training, and the music and compositional styles of Bach, Beethoven, Mozart, and Schubert, among others, were influences that would later become a motif in much of Ilaiyaraaja's compositions (such as the use of counterpoint). Ilaiyaraaja's classical music training culminated in him completing a course with a gold medal in classical guitar (higher local) with the Trinity College of Music, London.