Transgenic fish in development for aquaculture could escape from farms and interbreed with wild relatives in the nearby environment. Predicting whether escapes would result in transgene introgression is a major challenge in assessing environmental risks of transgenic fish. Previous studies have simulated gene flow from transgenic fish using mathematical modeling of fitness traits to predict the relative selective value of transgenic genotypes. Here, we present the first study of gene flow over the full life cycle in openly-breeding populations of transgenic animals, along with measurement of fitness traits. We conducted two invasion experiments in which we released two lines of growth-enhanced transgenic fish (T67 and T400), Japanese medaka (Oryzias latipes), into populations of wild-type (W) medaka in structured mesocosms. After several generations, the frequency of transgenic fish varied across replicates in the first invasion experiment (6 months), but the frequency of transgenic fish decreased in the second experiment (19 months). We also measured selected fitness traits in transgenic and wild-type medaka because these traits could be used to predict the relative selective value of a genotype. We found that: T400 males were more fertile than W males; offspring of W females lived longer than those with transgenic mothers; and W and T67 females reached sexual maturity sooner than T400 females. In contrast with other research that reported larger transgenic males had a mating advantage, we found that W males obtained more matings with females than T males; genetic background effects may account for our differing results as we compared W and T fish derived from different strains. The decreasing frequency of transgenic fish in the second invasion experiment suggests that transgenic fish had a selective disadvantage in the experimental environment. Our finding of transgenic advantage of some fitness traits and wild-type advantage in others is consistent with our invasion experiment results.