I know very little about Latin grammar but from what I am able to gather, magnum is the Latin neuter singular nominative adjective form that means “great” in English, magnus is the Latin masculine singular nominative form for that adjective, and magna is the Latin feminine singular nominative form for that adjective. The Latin noun opus, on the other hand, means “work” in English.
Since opus is neuter in Latin, the correct phrase for a “great work” or “masterwork” in the form of, say, someone’s musical composition, artistic creation, or novel is therefore magnum opus; in the strict literal sense, it should neither be the masculine magnus opus nor the feminine magna opus. In actual usage of the Latinate phrase by some English writers, however, magnum opus and magnus opus would sometimes be used interchangeably. This seems to me an indication that for English writers not knowledgeable of the declensions of Latin forms, the default usage for the adjective in that Latin phrase is the masculine magnus.