Norfolk Southern’s CEO will be under more pressure to improve profits after the railroad’s shareholders voted Thursday to elect three of the board members an activist investor nominated, but he won’t be fired right away.

Ancora Holdings had nominated seven directors as part of a bid to take control of the railroad’s 13-member board and overhaul its operations. The key support Ancora picked up from major investors, two major rail unions and proxy advisory firms wasn’t enough to persuade shareholders to elect Ancora’s entire slate.

Ancora’s Jim Chadwick blamed passive investors for failing to support the investors’ nominees. Chadwick promised to hold CEO Alan Shaw accountable and keep fighting to improve the railroad.

“For the passive investors. If anything should go wrong here and there’s another derailment and people die, this is on you,” Chadwick said. “You ignored the recommendation of the proxy advisors, the unions, the largest customer of the company. You gave us literally no support and we still won three board seats without you. What happens at Norfolk Southern now is on your firms and your conscience.”