All posts by yhong


By Yujie Hong | DDes’19,  Harvard GSD

The rail is too steep to climb (visual materials of this gif are from YouTube)

– Idea/Vision

My aspiration in this project is to propose a conceptual framework for augmenting designers’ intellect in design decision-making, by integrating VR/ AR technology and tangible inputs/outputs. I have been exploring the topic of applying AR/VR technology in architecture field for a long time. Taking this class really helped me narrow down my focus into some subareas. Partially inspired by the AfterMath project, the subarea I decided to pursue is assisting design decision-making. One proof-of-concept prototype is Structure.Ed, an AR+Lego toolkit that assists children to gain knowledge about structural engineering. After playing around with lego bricks, children can point phone camera at the lego structures they just made and the AR phone app will visualize potential consequences of assumed actions over them (eg. see if a bridge structure is stable or not when some heavy trucks run through it). I didn’t end up spending too much time in finalizing the implementation as I think developing the vision is more important and it provides a good guideline for future prototypes design.

– Motivation

In design process, architecture psychology is widely concentrated on “how will it look in actual?” Designers always think about their design and dream about how will it look after construction/ fabrication. This question usually makes a void for a very long period of time. But VR/ AR can give them all the answers they are looking for. It will connect a bridge between the present design and finished constructions, which might happen years after. In my master thesis project at Harvard, I did a user study to test architectural designers’ perceptions of scale in VR vs. in Rhino. The result shows that designers have much better estimations about scales of objects in virtual reality rather than in Rhino. As mentioned in Doug Engelbart’s conceptual framework, “advances in computing power, simulation, sensors, data analysis and machine learning make it highly plausible that computers will assist our decision-making process by providing a reliable estimate of the future-given situational and behavioral contexts of user interactions”. That is to say, as more information related to the problem has been collected, we can approach the problem more easily. By augmenting architects’ intellect in scale perceptions using VR/ AR, I believe they would feel more confident in making design decisions.

– Background Work (scientific or theoretical support)

Structural Engineering is a discipline that includes all the engineering knowledge related to the design of building. It is widely taught as a course in most architecture schools. Structural Engineering involves physics laws and empirical knowledge of the structural performance of different materials and geometries. “Structural engineers are trained to understand, predict, and calculate the stability, strength and rigidity of built structures for buildings and nonbuilding structures, to develop designs and integrate their design with that of other designers, and to supervise construction of projects on site. ”

“Structural engineers often specialize in particular fields, such as bridge engineering, building engineering, pipeline engineering, industrial structures, or special mechanical structures such as vehicles, ships or aircraft.” “Throughout ancient and medieval history most architectural design and construction was carried out by artisans, such as stone masons and carpenters, rising to the role of master builder.” “The physical sciences underlying structural engineering began to be understood in the Renaissance and have since developed into computer-based applications pioneered in the 1970s.”

Currently, there are many architecture firms and institutes investing a lot of resources and energy into the field of VR/AR. Most applications are designed as architectural space walk-throughs for facilitating architect-client communications. Interactive-wise, there are some interesting interior design tools for people to customize a virtual indoor space. However, AR/ VR education of structural engineering is a relatively under-explored research area.

From my perspective, the nature of structural engineering discipline involves physics knowledge (which can be calculated by mathematical equations like what have been explored in the AfterMath project) and empirical knowledge (involves predicting if certain actions like cracks will take place or not). It makes structural engineering education a perfect application area for AR-assisted design decision-making research.

– Related Work

Compared to virtual reality, augmented reality or mixed reality show greater potentials in construction projects. For example, employees at Gilbane Building Company, a commercial construction firm based in Rhode Island, now use HoloLens in construction sites. By overlaying the virtual models in the real site, you can look around and walk in the virtual design so you can check the pipelines and structures. It helps the company to notice real-world planning problems before building materials are wasted. ClayVision is an AR application that applied to urban scale. The system dynamically recognizes buildings in the city and transforms them into better-designed versions. It provides people with different experiences in the newly “elastic” city. Using a situated see-through display, HoloDesk allows users to manipulate virtual 3D models as if they are real objects. You can even use real object as a digital input to interact with the virtual models.

– Design and Implementation

Structure.Ed is an AR+Lego education toolkit for children to gain basic knowledge of structural engineering. First, children assemble a structural using lego blocks, for example, a bridge. Second, children can point the camera at the lego structure and the AR app will visualize potential consequences after simulating some trucks and cars running through this bridge. Third, designers may take advantage of the well-designed structure design as opposed to unstable structures in their design decision-making for real construction projects.

This proof of concept prototype displayed in this video is composed of GIFs I made by grabbing some visual materials from YouTube. But it demonstrates how the application can be used. For future implementation concerns, I plan to use Vuforia AR SDK and Unity 3D, which are all tools that I’m already familiar with.

– Usage Scenario

Assume an architect has designed a construction plan and he wants to make several design decisions to avoid problems. He can make a miniature model and points the AR camera at the model. He may changes the scale of the digital overlays to view his design in different sizes. And he can “modify” certain structures (like in the ClayVision project) to see if there is improvement. He may run simulations to see if the pipelines are well connected and circulated. And after he makes sure of everything, he can apply the modified version of design plan into real construction, to make a real bridge, road or building.

– Conclusion and Future Work

In this final project, I propose a conceptual framework called Augmenting Designers’ Intellect in Design Decision-Making by AR Technology. I made a proof-of-concept prototype that utilizes AR technology to visualize potential consequences of different structural design plans. It can also be treated as an educational tool for teaching this type of knowledge. I have shown the video demo to many friends and I also posted it on social media. I have received many positive feedbacks. Some people who work in the education business industry think it could be a very promising application if it is actually made and used in K12 education.

– Citations/Bibliography

Sang-won Leigh, and Pattie Maes. AfterMath: Visualizing Consequences of Actions through Augmented Reality. CHI 2015 Work-in-progress, 2015.

Takeuchi, Y., Perlin, K., ClayVision: the (elastic) image of the city, In Proc. CHI ‘12, 2012.

Zoran, A., Shilkrot, R., Paradiso, J., Humancomputer interaction for hybrid carving, In Proc. UIST’ 14, 2014.

Hilliges, O., Kim, D., Izadi, S., Weiss, M., Wilson, A., HoloDesk: direct 3d interactions with a situated seethrough display, In Proc. CHI ‘12, 2012.