Rubinstein's political career began when he founded Shinui after the Yom Kippur War. Shinui joined Yigael Yadin's Democratic Movement to form Dash. In the 1977 elections, Dash won 15 seats in the Knesset. Dash's victory came at the expense of the Alignment; for the first time in the 29 years since the founding of the modern state of Israel, the right-wing formed the government. However, Rubinstein opposed Dash's participation in Menachem Begin's Likud government coalition, and Shinui broke away from Dash. Rubinstein retained his seat in the 1981 elections, though Shinui was reduced to two seats. After winning three seats in the 1984 elections Shinui were invited into the governing coalition, and Rubinstein was appointed Minister of Communications. Rubinstein was re-elected again in 1988, but Shinui were left out of the government.
As an Education Minister, Rubinstein lowered the bar for high school graduates to enter higher education and developed a system whereby high school students would be required to take fewer matriculation exams: the subjects for the exams would be chosen each year by lottery. He also spoke out against the standardized tests which are required of Israeli university applicants (roughly equivalent to the SAT exams), claiming that if he had been required to pass these exams, he would not have been accepted to Law school.
Following Likud's victory in the 1996 elections, Rubinstein and Meretz left the government. He was re-elected for a final time in 1999, and resigned from the Knesset at the end of October 2002.
Rubinstein lived to hear his own obituary read in 2000, when due to a practical joke, Knesset speakerAvraham Burg was led to believe that he had died. Rubinstein, who was hospitalized at the time for a minor complaint, saw his eulogy broadcast on television.