Abstract: Cross-device federated learning (FL) enables a massive amount of clients to collaborate to train a machine learning model with local data. However, the computational resource of the client devices restricts FL from utilizing large modern machine learning models that requires sufficient computation. In this paper, we propose a federated layer matching algorithm that enables the server to build a deep server machine learning model from relatively shallow client models. The federated layer matching (FLM) algorithm dynamically averages similar layers in the client models to the server model, and inserts dissimilar layers as new layers to the server model. With the proposed algorithm, the clients are able to train small models based on device capacity, while the server can still obtain a larger and more powerful server model from the clients with decentralized data. Our numerical experiments show that the proposed FLM algorithm is able to build a server model $40\%$ larger than the client models, and such a model performs much better than the model obtained by the classical FedAvg, when using the same amount of communication resource.