What you are asking makes more sense now that you have described his seizures! It sounds like these may be atonic or "drop" seizures - is that correct? These can be quite dangerous because they come on without warning, but I can understand why you might want to take a more conservative approach to treatment, particularly since he is in a very safe and controlled environment. When a parent sees a child have a grand mal seizure you NEVER want to see that happen again, and that changes the risk/benefit analysis you bring to things like medication side effects. But since the episodes are so mild now, I think you can afford a more deliberate approach.
As AlohaBird said "brain damage" is not going to happen because your son has a few more small seizures! There are catastrophic epilepsy syndromes that can cause major brain damage, and prolonged (think 20-30 minute) and recurrent seizures can also cause issues - but what you have described doesn't sound anything like that. So please don't let that scare you into making a decision you are not comfortable with.
Make sure you have as much information as you can get. Pursue some metabolic testing to see if that reveals anything. Find out from your neurologist if there is anything in your son's medical situation that in their clinical experience makes it more likely that his seizures will progress and worsen. In our case, the experienced doctors at CHOP felt that the way my daughter's initial seizures presented were generally the "beginning of a pattern" when seen in kids her age - so even though technically 3 seizures in one day does not mean an epilepsy diagnosis and trigger medication, he recommended treatment. It turns out that they were right about that. I wish that her doctors had taken the time to spell this out more clearly at the time - so see if you can get them to slow down and EXPLAIN their thinking to you.
If things do worsen and he begins to have more frequent or more serious seizures (hopefully not!!), or if you are convinced for other reasons that it is important to begin treatment now, don't be paralyzed by reading the list of possible medication side effects. While I don't want to downplay medication side effects, particularly for AEDs, not everyone experiences them. Many many people have no issues whatsoever. My daughter had a terrible time on Keppra, but is doing quite well on her current medication (Vimpat).
And you do have other options, as AlohaBird has mentioned! For a variety of reasons, in my family and for my daughter's specific needs we have not pursued a dietary approach to epilepsy treatment - but it really is a proven treatment option that you can consider. And it has been shown to be particularly effective in children your son's age. There is a wealth of information on this site, and you should definitely ask your doctor about it.
These are difficult decisions. One of the things that I realized early on is that epilepsy is so variable, and the course and prognosis of the illness is so individual, that figuring out the best course of action can be very challenging. I wish you all the best as you navigate this world! Please keep us posted.