Can adverbs and adjectives both be used to modify gerunds or gerund phrases?
By definition, gerunds and gerund phrases function grammatically as nouns. As such, they can only be modified by adjectives and not by adverbs.
In the first sentence you presented, “I regretted his leaving the job,” the gerund phrase is “his leaving the job.” The pronoun “his” is integral to that gerund phrase, which, of course, is the direct object of the verb “regretted.” Although internally, “his” functions as modifier of the phrase “leaving the job,” it can’t be considered as an adjective modifying the gerund phrase “his leaving the job.” It’s actually part and parcel of that gerund phrase and can’t be considered as a separate grammatical entity.
In the same token, in the second sentence you presented, “He enjoys jogging three miles during evenings,” the phrase “during evenings” is integral to the gerund phrase “jogging three miles during evenings” as direct object of the verb “enjoys.” The phrase “during evenings” does function internally as an adverbial modifier of the verb “jogging,” but it is part and parcel of that gerund phrase and can’t be considered as a separate grammatical entity acting on the gerund phrase. From the standpoint of the entire sentence, therefore, “during evenings” is not an adverbial modifier.
So my answer to your question, “Can adverbs and adjectives both be used to modify gerunds or gerund phrases?”, my answer is only internally within the gerund phrase itself but never externally. We can modify a gerund or gerund phrase internally with adjectives and adverbs in any number of ways, but once we have done so in a particular sentence, the modifications become integral to the gerund phrase in its function as a noun—whether as subject or object in the sentence.
Of course, a legitimate grammar question regarding gerund phrases is this: Can a gerund phrase as a distinct, established grammatical entity—whether as subject or object—be modified at all in a sentence? I don’t think so. In a sentence, a gerund phrase as a grammatical entity is already cast in stone, so to speak. Any modification in a sentence that uses a gerund phrase can only be done with the operative verb (and also with the subject, of course). For instance, the sentence “I regretted his leaving the job” can be restated as, say, “I terribly regretted his leaving the job” or “I regretted terribly his leaving the job,” but in both cases, the adverb “terribly” isn’t modifying the gerund phrase “his leaving the job” but the verb “regretted” instead. If we put “terribly” at the tail end of the sentence, “I regretted his leaving the job terribly” (with the intention of making the adverb “terribly” modify the gerund phrase “his leaving the job”), the syntax of the sentence becomes defective. Indeed, “terribly” becomes a dangling modifier. Being an adverb, it obviously can’t modify the noun “job” and, being in a wrong position, it is hard put as well to find a verb that it can modify properly.
We can similarly restate the sentence “He enjoys jogging three miles during evenings” as, say, “He immensely enjoys jogging three miles during evenings.” Here, the adverb “immensely” clearly modifies the verb “enjoys” and not the gerund phrase “jogging three miles during evenings.” In this case, in fact, we find that “immensely” can only take that position and nowhere else in that sentence. We can't construct the sentence as “He enjoys jogging three miles during evenings immensely,” where the adverb “immensely” is clearly a dangling modifier. This is an unmistakable indication of the function of “immensely” as a modifier of the verb “enjoys” rather than of the gerund phrase “jogging three miles during evenings.”
I might add that even if we reduce that gerund phrase to its limiting gerund form, “jogging,” it still can’t be legitimately modified by the adverb “immensely.” Why? It’s because whether we construct the sentence as “He immensely enjoys jogging” or as “He enjoys jogging immensely,” the adverb “immensely” still modifies the verb “enjoys” and not “jogging.” This is further validation that gerund phrases, being nouns in function, can’t be externally modified by adverbs at all.
