We clearly saw that there is some sort of connection between Shanks and Gorosei. Gorosei clearly stated that they do not have time right now because of the Reverie Conference, but still they managed to give time to Yonko Shanks to address his concern. Gorocei are the five elder stars of the World Government, so it makes sense that they’re the busiest right people in the world right now. Despite all that, the Gorosei allowed Shanks to enter, and make his request.
They are at the topmost position in the World Government. One could even say that they are the ones who control the policies and the way the whole of the World Government is actually run. Now, the interesting thing here to note is that Shanks is one of the four Yonko. Yonko are the pirate crews that control the seas. These are Kaido, Big Mom, Blackbeard and Shanks. So, Yonko and Gorosei are like the opposite sides of the forces trying to control the seas. You can say that the Yonko are in direct opposition to the World Government, and the Government uses their military, called the Marines, to try to take them down.