I would say it's the verbs because you have to deal with mood, tense, number, voice, and conditionality to figure out their correct form and usage in particular statements. You are therefore on the right track in restudying them. The next tough part would be the English idioms and figures of speech. There are thousands of these forms, and they very often deviate from or subvert the grammar conventions and rules you have been learning so assiduously. To be really fluent in both written and spoken English, in fact, you need to memorize many of these tricky forms so you can use them correctly and confidently when the situation demands.