Different style books vary in their approach to making plurals of abbreviations and numbers. Some add an apostrophe before the lowercase s— CEO’s —and others prefer to omit the apostrophe— CEOs.
I vote with the latter group because the less punctuation used, in general, the clearer the text. So I recommend writing in the following style:
However, sometimes an apostrophe is necessary to avoid confusion. What if you want to count the number of times the first letter of the alphabet is used in a paragraph? If you write that you saw nine as, your readers are left scratching their heads. Nine a’s clarifies what you meant.