Ali Akbar Khan was born on April 14, 1922 in the village of Shibpur, Comilla, in present-day Bangladesh (then East Bengal), to Baba Allauddin Khan and Madina Begum. Khan began his studies in vocal and instrumental music at very early age under his father. He also studied drums from his uncle, Fakir Aftabuddin. Allauddin Khan trained him on several instruments, but decided finally that he must concentrate on the sarod. After years of rigorous training he gave his debut performance in his mid-teens. Khan became the court musician of the royal family of Jodhpur at the age of 22.

Khan performed all over India to great applause and rave critical acclaim, and also traveled the world extensively in the West. In 1956, Khan founded the Ali Akbar College of Music in Calcutta, with the mission to teach and spread Indian classical music. Two years later, he founded another school of the same name in Berkeley, California; it moved to its present location in San Rafael, California in 1968. Khan has been based in the United States since the foundation of the San Rafael school, although he tours extensively. However, ill health in recent times has curtailed this. In 1985 he founded another branch of the Ali Akbar College of Music in Basel, Switzerland.

He is a past master at outlining a melody with great economy of stroke, which has stood him in good stead in his short 78 rpm records in the middle of the last century. His long concert performances progress from the meditative (alap, jod) to the exhilarating (gat, jhala) in a highly structured build-up in the Senia beenkar style. He is also a fine exponent of "sawal-jawab", a dialogue between two instruments (usually one melodic and one percussion). Of late, ill health has reduced the frequency of his concerts and affected his physical dexterity on his instrument.

Khan has participated in a number of classic jugalbandi pairings, most notably with his fellow-student and former brother-in-law Ravi Shankar, the late Nikhil Banerjee and the violinist L. Subramaniam. A few recordings of some spectacular duets with Vilayat Khan also exist. He has also collaborated with many well-known Western musicians.

Khan has been awarded the Padma Vibhushan in 1989, one of the highest civilian awards in India, as well as a plethora of other awards and honours. In 1997, Khan received the National Endowment for the Arts' prestigious National Heritage Fellowship, the United States' highest honour in the traditional arts. This followed a MacArthur Genius Grant in 1991. Khan has also received several Grammy nominations.

Khan's son Ustad Aashish Khan (Grammy nominee of 2007) is also a reputable sarod player