The construction industry claims one of the highest occupational fatality rates. A great portion of onsite accidents is attributed to the unsafe and risky operations of heavy construction equipment, such as excavators. Such risky operations stem in many cases from inadequate training of heavy equipment operators. Given the high cost of training programs, where operators are trained on actual equipment, Virtual Reality (VR)-based training simulators are gaining popularity both in the industry and in academia. However, current simulators are limited in the available training scenarios and in capturing the dynamic interactive environment where several operators need to work in tight areas while ensuring smooth and safe operations. In the available simulators, training scenarios are developed manually and therefore there are usually a limited number of scenarios from which the trainees can choose. While VR environment is investigated for application in construction training simulator, advanced immersive technologies that can provide a more realist training experience are not fully considered. Additionally, while the trainees are able to control, navigate and operate the VR equipment, the realistic behavior of surrounding pieces of equipment, with which the operator must interact, is scarcely simulated. Last but not least, the existing simulators are mainly designed to accommodate a single user at a time. Given that communication is an indispensable ingredient for effective and safe equipment handling, training simulators need to address multi-user environment where several trainees can simultaneously work, or collaborate, on a training scenario. In this paper, a novel conceptual framework is proposed to develop the training scenarios form actual construction sites by integrating sensory data captured from the site and an actual spatial model (including the Building Information Model and the terrain model). Additionally, it is proposed to develop a Multi-Agent System inside the training environment to simulate the behavior of multiple pieces of equipment surrounding the virtual equipment controlled by the trainee. These components are bound together in a multi-user immersive environment. A case study is conducted to demonstrate the feasibility of the proposed method. It is demonstrated that the proposed framework is providing a promising alternative to the current training simulators where the operators can be economically subject to a multitude of realistic training scenarios to develop both their operational and communication skills required for safe and productive operations.