Frames and plastics are part of the window... A perfectly clean piece of glass in a mucky filthy frame looks terrible.

Frames and plastics are part of the "window"

Of course many frames do not come out perfectly white, but that is still no reason to not make them look as good as possible.

I make a habit of cleaning frames with windows and by maintaining the frames the glass actually becomes easier to keep clean over time as window glass often gets some of its' spots and dirt buildup from the frame itself, usually the top.