Why Most Star Trek Aliens Look Mostly Human
Plot points, money, and costume creation
There are actually several reasons that the majority of the aliens found in the first few series, at least before we make it to the gamma quadrant, are somewhat humanoid in nature. This has to do with both the origin of the quadrant and the forces that have long protected it. The first reason is that the majority of the races in Trek, from Cardassians to humans, were genetically modified by the sewing of DNA onto their planets by a group of entities called the Progenetors. While little is known about them, they were advanced enough to leave behind a hidden code in the DNA of the races that they seeded that would reveal a hologram when combined. Before this point in time, there were no other humanoid races, and it is heavily implied that only this DNA is keeping the current races even slightly related. To read more about the hologram, Progenetors, and other speculation about them you can check their memory alpha articles.
Next is the fact that the same race and some others then decided to protect and shepherd the galaxy. They went so far as to exterminate some other races, some even thinking the Iconians were a victim of their crusades. Protective installations such as shields, dampening fields, and even evolved creatures can be found through the various different planets within their range. This points to them having kept non-similar lifeforms from evolving or to them being exterminated as the number of M-class planets without any current lifeforms on them but that show previous signs of civilization is very high.
Another form of protection is that number of races that have outright stated they maintain a humanoid look to keep from being noticeably different or upsetting humanoids. A number of times we see energy based beings who literally disappear off of ships at their own will. Finally, keeping the mostly non-humanoids out until the Xenosphore, Seed, Pah Wraiths, creatures in fluidic space, and even Troi's child, allows them to be seen as truly special. Each non-humanoid presents a medical, spiritual, and intellectual quandary that allows more exploration.
Out Of Universe
First of all, getting someone to play a non-human is pretty hard. Even though there are willing actors, stories abound about prosthetics, paint, and costume elements that do nothing more than make it more difficult for people to move. Second, it's expensive to do that kind of work. A one-off character who is done with CGI or who needs to be edited out could cost more than the entire rest of the episodes production. This makes it make sense to save them for a more special point in time and to use them to be more impact than anything.
Finally, Trek is an extremely old show, that has grown leaps and bounds. From the first sets that didn't even have moving buttons to now, the design and history behind it has changed greatly. Asking costume designers in the early 70s to come up with non-human entities for a relatively low budget show just wasn't going to happen. In the end, we have a ton of relatable aliens, some that stretch the bounds of our imagination, and a firm idea of where character design is going or can go in the series. With the teasing of more humanoid aliens for the Picard series, we can see that Trek will stick to its roots in some ways, but we might be lucky enough to see it not do so in other ways.