Cyber-physical systems (CPSs) are designed to integrate computation and physical processes through constantly interacting with the physical environment. The complexity and uncertainty of the environment often come up with unpredictable situations, which place high demands on the dynamic adaptability of CPSs. Further, as the environment evolves, the CPS needs to constantly evolve itself to adapt to the changing environment. This paper presents a research plan that aims to develop a novel framework to address CPS design challenges under uncertain environments. We propose to utilize evolutionary computation and reinforcement learning techniques to design control policies that can adapt to the dynamic changes and uncertainties of the environment. Further, novel testing and evaluation approaches that can generate test cases while adapting to dynamic changes in the system and the environment will be explored.