Leveraging Human-Robot Collaboration in Construction

Research Overview

Observed Problem:

Current construction robots rely on bulky industrial robotic arms and simplistic control leading to unsafe operating conditions and limited capabilities that do not leverage human expertise. Given this scenario, we propose to rethink construction tasks using compact, compliant, safe, and multi-purpose robots enabling collaboration between humans and robots.

Primary Research Objective:

Develop a haptic simulation environment to explore lightweight, multi-robot collaborative systems with human interaction for five hazardous and strenuous construction tasks. The results will be experimentally validated.

Potential Value to CIFE Members and Practice:

• Rethink hazardous construction tasks using compact, safe, multi-purpose robots and human-robot collaboration.
• Collecting production data and motion capture of construction workers to learn from construction experience.

Research provides relevant insights for:

Corporate managers involved with strategic technological innovation in construction practice and researchers in the robotics field interested in exploring human-robot collaboration in an unstructured environment.

Research and Theoretical Contributions

• Develop a haptic simulation environment to explore light-weight, multi-robot collaborative systems with human interaction for the five hazardous and strenuous tasks.
• Measure adaptability to workplace situations and production efficiency (e.g., height limit, component sizes, payload, speed, and space).

Industry and Academic Partners:

Goldbeck

Research Updates & Progress Reports

Progress Report - April 2020

Construction tasks overview

Based on observation from Goldbeck’s site visits we defined three dangerous, repetitive, strenuous, and unpredictable manual construction tasks to study:

1. Bolting of steel beams to steel columns (Figure 1).
   
   On a six-level parking garage, workers install 380 beams with four bolts on each side. The task consists of fitting the beam between the columns by three workers and then two workers bolt the beams to the columns. We gained access to production data on-site tracking the bolting cycle time per each side of the beam and the preparation time per day.
2. Welding plates for structural columns (Figure 2).

On a six-level parking garage, one worker installs 144 anchors. The task consists of placing the anchor, fixing the anchor with welded dots, and welding the four sides of the plate. We gained access to production data on-site tracking the cycle time per anchor and the total preparation time per workday.

3. Placing concrete joints (pouring, shot blasting, and coating) (Figure 3)

On a six-level parking garage, 6000 linear meters of concrete joints are installed by three different crews. The first crew of two workers pours concrete with a concrete pump and nozzle that first the joint width, the second crew of one worker shot blasts with a shot blasting machine, and the third one consisting of four to five workers manually provides a three-layer coating. We gained access to production data on-site tracking the cycle time of the three subtasks by linear meter and the total preparation time per workday for the three subtasks. Additionally, we have detailed data from a Veidekke construction site following four workers installing interior drywall panels.

4. Drywall placing (Figure 4).

The tasks observed during the course of a week included layout, the first side of drywall with studs, electrical installation, and the second side of the drywall.

Robotization of the Construction Tasks

 

Following Bernold’s hierarchical structure of construction activities as a guideline to evaluate their potential for robotization, we classified the four tasks following six hierarchy levels: organization, project, activity, process, work task, and motion (Figure 5). The first three levels were evaluated by Bernold as unique and non-repetitive activities, while the process, work task, and motion were considered semi-repetitive and repetitive, and thus optimal for on-site robotization.

We could, for example, characterize welding as the sequence of placing anchors, fixing anchors, trace the anchor’s edge, and welding. Each of these tasks could be decomposed into several motions such as grab, put, align, and push. To decompose these complex processes into repetitive tasks we should analyze in detail the present construction method, the robotic alternatives, the components for each option, and the process to automate the construction task (Warszawski,1989).

In a second level, for each of the motions that make up the tasks, we can define necessary manipulation skills, sensors, dynamics, and controls (Figure 6). This information will be used to design light, compliant, and collaborative robot solutions with haptic simulations and then test the solutions with a physical prototype (Figure 7).

The Bolting example will test a two-hand manipulation mechanism in haptics simulation. We have obtained detailed 3D models of the beams, columns, bolts, and steel angles, which we have imported to SAI. Additionally, Goldbeck has manufactured a physical prototype to validate the simulations. The validation tests will make use of the existing KUKA robot arms available in the Stanford Robotic Laboratory (Table 1). This seven DOF arm has integrated sensitive torque sensors in all seven axes, contact detection capabilities and programmable compliance. It can handle the manipulation of fragile and sensitive objects and has contact detection capabilities that enable humans to directly collaborate with the robot. This robot can learn by demonstration, guided by a human to a desired position and saving the points along the trajectory path in the robot program. We can also teach this robot through haptics simulations of the motion and physical interactions in SAI.

---

Progress Report - December 2019

Overview & Observed Problem

Robotic applications are being developed and tested on construction to handle heavy, repetitive, and hazardous tasks like drywalling, bricklaying, and welding. These robots mostly rely on bulky industrial robotic arms and overly simplistic control algorithms, leading to unsafe operating conditions and limited capabilities that do not leverage human expertise while performing the construction task on site, other than in the role of an operator that oversees robotic performance.

In addition, the techniques and products used in construction (from brick sizes to gypsum board dimensions) come from a long history of development and traditions based on labor-intensive processes. Efficient robotization may require rethinking some of these methods and standards (Bernold, 1987). By replacing the human worker with a single task machine, we could be overlooking what is the best possible product and process for the construction task.

Theoretical & Practical Points of Departure

Construction task automation:

Previous research has evaluated which operations are the most suitable to be robotized and automated. According to Bernold (1987), manufacturing achieved mass production with intensive research and restructuring of their traditional methods to make them more amenable to automation. At that time, both the construction industry mindset and the available technology were not ready to efficiently identify and automate the repetitive tasks that take part in construction projects (Bernold, 1987).

Kangari (1985) and Skibniewski & Nof (1989) studied repetitive construction operations by breaking them down into individual processes, tasks, and sub-tasks. Similarly, Bernold proposed a hierarchical structure of construction activities as a guideline to evaluate their potential for robotization following six hierarchy levels: organization, project, activity, process, work task, and motion. The first three levels were evaluated as unique and non-repetitive activities, while the process, work task, and motion were considered semi-repetitive and repetitive, and thus optimal for on-site robotization (Figure 1).We could, for example, characterize earthmoving as the sequence of loading, dumping, and traveling tasks. In turn, each of these tasks could be decomposed into several motions such as grab, put, turn, and push. To decompose these complex processes into repetitive tasks we should analyze in detail the present construction method, the robotic alternatives, the components for each option, and the process to automate the construction task (Warszawski,1989).

According to Skibniewski and Nof (1989) and Warszawski (1989), the value of robotization of tasks increases when these happen in high altitude, extended periods of overtime work, or under any hazardous condition. Some of these tasks are structured enough to be autonomously performed by a robot, while others must still rely on the capabilities of human workers (Khatib, 1998).  The unstructured and dynamic environment of the construction field poses considerable technical challenges (Kangari, 1985) and, even with today’s technologies, construction robotics mostly consists of industrial arms mounted on moving platforms that perform a single task (Ragaglia et al., 2017; Usmanov et al., 2017; Yamada et al., 2017; Zedin et al., 2017). This configuration has led to complex setups and programming, non-versatile, bulky, stiff, and unsafe robots that do not leverage human collaboration in the field.

Inspiration from robotics:

State of the art robotics has progressed in mobility, autonomous manipulation skills, AI reasoning and planning, and physical interaction with unstructured environments through multimodal sensing and environment modeling. These capabilities are key to the application of robotics in space, underwater, domestic environments, medical settings, and construction (Khatib, 1998). These environments are challenging for robots because they are highly unstructured and dynamic, changing even while the task is performed (Groll et al., 2017; Kangari, 1985).

While the manufacturing industry has traditionally separated workers from robots for safety reasons, integration of humans and robots in the construction industry constitutes a key ergonomic study (Skibniewski & Nof, 1989). Ergonomic studies demonstrate the human intent and address how motions can be optimized on the basis of least energy consumption and safety (Bernold, 1989). To successfully introduce robotics in the human environment we need to develop practical, compliant, safe, and easy to use systems that are as reliable as the human worker (Khatib, 1998).

The use of bimanual robotic manipulation, despite its many advantages, is rare in industrial applications (Kempe, 2007; Smith, 2012). Nevertheless, previous research has provided useful implementation models of two-hand object assembly, where the objects held by the robot are increasingly constrained either with respect to the ground or with respect to each other. This gives rise to constraint forces which must be explicitly controlled to prevent damage to the object and control slippage. The virtual linkage model provides a mathematical foundation to control these constraint forces (Khatib, 1996; Williams, 1993). Furthermore, the augmented object model (Chang, 2000) can be used to describe the dynamic behavior of the two manipulators plus object system.

Motion coordination for multiple mobile robots and human-robot collaboration leveraging bimanual manipulation has been previously considered for drywall operations (Khatib, 1998). This approach incorporated a combination of autonomous behaviors and guided motion interactions for collaboration between humans and robots (Figure 2). The experiments showed the robots could improve the quality and reduce the strain required to perform the task manually, while the workers contribute their knowledge and experience (Khatib, 1998).

Humans and robots can also collaborate with a haptic-visual interface that combines the dexterity, flexibility, problem-solving, and expertise of humans with the strength, endurance, and precision of robots in the field. A successful example is Ocean One. This underwater humanoid robot successfully exploits collaboration between a robot and a human to execute tasks at depths where divers cannot reach. The human operator is able to feel exactly what the robot feels through a haptic interface used to guide and control the robot (Figure 3).

Figure 3: A human operator guiding the robot in an underwater bimanual task through the haptic-visual interface (Khatib et al., 2016)

Research Methods & Work Plan

The goal of this research is to rethink five repetitive and hazardous construction tasks (drywall placing, bolting, welding, painting, and shotcrete) using compact, compliant, safe, and multi-purpose robots that enable collaboration between humans and machines.

To accomplish this, we propose to:

• Extract context and production data from construction tasks on site.
• Develop a haptic simulation environment to explore light-weight, multi-robot collaborative systems with human interaction for the five hazardous and strenuous tasks.
• Measure adaptability to workplace situations and production efficiency (e.g., height limit, component sizes, payload, speed, and space).
• Iteratively improve the task design based on industry partners’ feedback.
• Validate the robotic process with physical experiments.

Simulation methods

The simulation environment, SAI (Simulation and Active Interfaces), developed by the Stanford Robotics lab in collaboration with Google, integrates the simulation of control and physical interaction. SAI allows its users to control and test robot’s behavior through haptic/UI interfaces and provides efficient and realistic simulations of complex environments (Khatib et al., 2004).This simulation environment was previously deployed in the OceanOne project to simulate underwater exploration with pre-loaded scenes. In the same way, we can load the project BIM as a scene to explore the robot interaction with the environment and construction products (Figure 4). 4D models and parametric design and schedules used in construction will help us (the research team) iterate on the robotized task design in order to improve the available processes and products.

We will base these virtual explorations of human-robot collaborations on thorough observations of task context, production information, and motion capture data (collected using Xsens devices) on construction sites for the five selected tasks: painting, shotcrete, welding, bolting, and drywall. Next, we aim to design bimanual haptic and autonomous controllers to simulate this data with a rich construction environment in SAI. Industry feedback on the simulations will help us iterate and develop compact multi-purpose robot configurations with human-robot collaboration and new construction products or processes better suited for robotic automation.

Expected Contributions to Practice

• Our contributions to the robotics field are exploring robot-human collaboration in an unstructured environment that provides a rich variety of tasks and expanding the understanding of grasp, use of tools, and manipulation skills.
• For the construction industry, our expected contributions are rethinking hazardous construction tasks using compact, safe, multi-purpose robots and human-robot collaboration, and collecting production data and motion capture of construction workers to learn from construction experience.

Expected Contributions to Theory

• Develop a haptic simulation environment to explore light-weight, multi-robot collaborative systems with human interaction for the five hazardous and strenuous tasks.
• Measure adaptability to workplace situations and production efficiency (e.g., height limit, component sizes, payload, speed, and space).

Publications

References

• Bélanger-Barrette, M. (2014). How do industrial robots achieve compliance. Retrieved from https://blog.robotiq.com/bid/69962/How-Do-Industrial-Robots-Achieve-Com…
• Bernold, L. E. (1987). Automation and robotics in construction: a challenge and a chance for an industry in transition. International Journal of Project Management, 5(3), 155-160.
• Chang, K. S., Holmberg, R., & Khatib, O. (2000). The augmented object model: Cooperative manipulation and parallel mechanism dynamics. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065) (Vol. 1, pp. 470-475). IEEE.
• Groll, T., Hemer, S., Ropertz, T., & Berns, K. (2017). A behavior-based architecture for excavation tasks. ISARC 2017 - Proceedings of the 34th International Symposium on Automation and Robotics in Construction, (Isarc).
• Kangari, R. (1985). Robotics Feasibility in the Construction Industry. Retrieved from http://www.iaarc.org/publications/fulltext/Robotics_feasibility_in_the_…
• Kemp, C. C., Edsinger, A., & Torres-Jara, E. (2007). Challenges for robot manipulation in human environments [grand challenges of robotics]. IEEE Robotics & Automation Magazine, 14(1), 20-29.
• Khatib, O., Yokoi, K., Brock, O., Chang, K., & Casai, A. (1999). Robots in human environments. Proceedings of the 1st Workshop on Robot Motion and Control, RoMoCo 1999, 213–221. https://doi.org/10.1109/ROMOCO.1999.791078
• Khatib, O., Yeh, X., Brantner, G., Soe, B., Kim, B., Ganguly, S., … Creuze, V. (2016). Ocean one: A robotic avatar for oceanic discovery. IEEE Robotics and Automation Magazine, 23(4), 20–29. https://doi.org/10.1109/MRA.2016.2613281
• Khatib, O., Yokoi, K. Chang, K., Ruspini, D., Holmberg, R., & Casal, A., (1996). Vehicle/arm coordination and multiple mobile manipulator decentralized cooperation, 2, 546–553 vol.2. https://doi.org/10.1109/IROS.1996.570849
• Khatib, O. (1998). Mobile manipulation: The robotic assistant. Robotics and Autonomous Systems, (26), 175–183.
• Skibniewski, M. J., & Nof, S. Y. (1989). A framework for programmable and flexible construction systems. Robotics and Autonomous Systems, 5(2), 135–150. https://doi.org/10.1016/0921-8890(89)90006-7
• Smith, C., Karayiannidis, Y., Nalpantidis, L., Gratal, X., Qi, P., Dimarogonas, D. V., & Kragic, D. (2012). Dual arm manipulation—A survey. Robotics and Autonomous systems, 60(10), 1340-1353.
• Ragaglia, M., Argiolas, A., & Niccolini, M. (2017). Cartesian-Space Motion Planning for Autonomous Construction Machines. In 34th International Symposium on Automation and Robotics in Construction (pp. 983–990). Taipei: International Association for Automation and Robotics in Construction. https://doi.org/https://doi.org/10.22260/ISARC2017/0136
• Usmanov, V., Bruzl, M., Svoboda, P., & Šulc, R. (2017). Modelling of industrial robotic brick system. In 34th International Symposium on Automation and Robotics in Construction (pp. 1013–1020). Taipei: International Association for Automation and Robotics in Construction. https://doi.org/https://doi.org/10.22260/ISARC2017/0140
• Warszawski, A. (1987). Application of Robotics to Building Construction by. International Council for Building Research Studies and Documentation.
• Williams, D., & Khatib, O. (1993, May). The virtual linkage: A model for internal forces in multi-grasp manipulation. In [1993] Proceedings IEEE International Conference on Robotics and Automation (pp. 1025-1030). IEEE.
• Yamada, M., Fujino, K., Kajita, H., Hashimoto, T., & Technology, A. (2017). Survey of the line of sight characteristics of construction machine operators to improve the efficiency of unmanned construction. In 34th International Symposium on Automation and Robotics in Construction (pp. 588–593). Taipei: International Association for Automation and Robotics in Construction. https://doi.org/https://doi.org/10.22260/ISARC2017/0082
• Zedin, T., Vitalis, L., Guéna, F., & Marchand, O. (2017). A method based on C-K Theory for fast STCR development: The case of a drilling robot design. In 34th International Symposium on Automation and Robotics in Construction (Vol. 34, pp. 464–471). Taipei: International Association for Automation and Robotics in Construction. https://doi.org/https://doi.org/10.22260/ISARC2017/0064

Original Research Proposal

Proposal 2019-05

Funding Year: 

2020