I’ve been looking for a framework that will allow me to start prototyping this project – trying to get to a Minimal Viable Product as soon as possible.
The above image illustrates the concept well. Henrik Kniberg (the creator of the above image) goes into more detail about the idea in this blog post.
Quoting from the original proposal for the project:
The aim of this project is to create a digital sculpture as big as India itself, accessible from anywhere in India by anyone with a smartphone – as well as people outside of India via the web.
It’s therefore essential to me that this work is viewable by as many people as possible. The web seems the logical place to do that! In addition, as the sculpture is three-dimensional, I know that I need a framework that can render that kind of information via websites. It turns out that things have moved on dramatically since the days of VRML, with three.js the de-facto library for displaying with 3D content in a standards compliant way.
Virtual Reality (VR) is a very hot topic of research and development at the moment, and with that in mind I wanted to make sure that whatever system I developed to display the sculpture would be compatible with as many VR platforms as possible. A-Frame is the perfect framework to do that:
- It’s built on top of three.js
- It’s compatible with not only the latest VR headsets, but also lower cost platforms such as Google Cardboard – and even degrades gracefully if someone views A-Frame content without a VR headset (such as via a desktop computer or mobile phone)
Digging deeper into A-Frame Frequently Asked Questions (FAQ) led me to discover that it uses the entity component system pattern – something that I had my attention drawn to some time ago from my brother, as well as via the Unity platform. As the FAQ states:
A-Frame is based on an entity-component-system pattern (ECS), a pattern common in game development that emphasizes composability over inheritance:
- An entity is a general-purpose object that inherently does and renders nothing.
- A component is a reusable module that is plugged into entities in order to provide appearance, behavior, and/or functionality. They are plug-and-play for objects.
- A system provides global scope, services, and management to classes of components.
ECS lets us build complex entities with rich behavior by plugging different reusable components into the sockets on the entity. Contrast this to traditional inheritance where if we want to extend an object, we would have to manually create a new class to do so.
ECS grants developers the key to permissionless innovation. Developers can write, share, and plug in components that extend new features or iterate upon existing features.
Having previously used an object orientated programming while using openFrameworks, I’m excited to try this new approach with this web-based project.
This is how I described the sculpture in the original proposal to the British Council:
Imagine a virtual layer of silk as big as the subcontinent, seeming to float in the sky above..
But what is the best method of displaying this digital silk? Voxels à la Minecraft? After disappearing down a rabbit hole of the differences between voxels and pixels, a fantasy console, dynamic lighting for images, voxel drawing programs and the mathematics behind computer graphics, I started searching for ways of making an imaginary map that could sit atop India.
I found a Javascript library for displaying mobile imaginary maps and a brilliant article on creating fantasy maps, but then started thinking about how to make the form of the sculpture – I didn’t want to make something random, but I wasn’t sure how to make a form on the scale of an entire continent.
Again referring back to my original proposal:
Users in the same physical location will be able to see other people’s interactions with the surface from the same location – they’d be able to raise or lower their part of the surface in real time by simply clicking their mouse or touching their mobile phone screen – making a new digital landscape above the physical geography of India.
I realised I already had the answer – the topology of India itself would be the starting point for the sculpture, which would then be altered by the actions of users in real-time.
How could I get the entire landscape of India in a digital format? I quickly discovered Earth Explorer from the US Geological Service, but found the user interface clunky to say the least – and which dataset is the best? The Shuttle Radar Topography Mission was the consensus opinion. Google provides a way of getting elevation data for any point on Earth, but I knew I didn’t want to have to rely on an external server for this starting point – I wanted to sculpt it myself and allow users to sculpt it too. Thanks to Patricio, I saw a recent announcement by Amazon and Mapzen (the company he works for) around open sourcing global elevation data.
After finding an open source Javascript 3D world viewer and re-visiting the brilliant OpenStreetMap, I realised that it was important to stick to the MVP principle and see what was available in the existing A-Frame/three.js ecosystem.
One of the best things about three.js is it’s quantity of examples – allowing users to quickly see the possibilities of the platform. Whilst browsing the examples, I found this “Geometry Terrain” and this “Dynamic Terrain” that reminded me of my original proposal, but inverted – i.e. in the skies above.
There are a variety of articles on how to create three.js visualisations of real world landscapes, but was there anything up to date and available for A-Frame?
Browsing a handy list of awesome things made using A-Frame, I discovered a component for making a heightgrid, which looks like a great start – as well as a component for taking terrain data, with several examples.
VR is all very well, but this project requires something different – Augmented Reality (AR). AR has gathered mainstream attention recently with the release of Pokémon Go as well as attention in the world of tech startups – especially around Magic Leap.
Creating mobile AR with A-Frame would require the combination of live video from a mobile phone camera with three-dimensional imagery super imposed on top. At luck would have it, whilst being “lazy like a fox” and browsing through the A-Frame blog led me to discover this project, by Dietrich Ayala:
Unfortunately, the example is only compatible with Firefox on Android, but it’s certainly a great place to start.
This got me to thinking not only about the number of mobile phone users in India, but also how many of those use smart phones and how many of those users were using Android.
There are 1,034,253,328 mobile phone users in India, with 29.8% owning smartphones and 97% of those using Android. I make that a potential audience of 298,961,267! Naturally, not all of those will meet the minimum hardware requirements for A-Frame.
With the choice made on the front end for the MVP, I started to thing about the backend of the system. My priority was a backend system that could had a thriving community, would allow real-time interaction and would be able to deal with Geospatial data in a sensible way.
I starting thinking about making a system using Python that would allow for real-time interactions:
The idea was to use WebSockets to update viewers landscapes as other users interacted with them.
After taking some advice from Szymon, Igor and Patricio, I decided to plump an MVP built-in Node.js as it was the system that was already being used for several A-Frame components and had a thriving community around it. Both Patricio and Szymon recommended that I use PostGIS as a database platform as it was designed to deal with Geospatial data and relations within the database engine.
In summary, the first pass at this system will consist of the following platforms and components:
- A-Frame, including the HeightGrid component, the Terrain Model component and the AR experiment.
- Node.js, including the Express framework, socket.io for WebSockets, Twilio for SMS interaction and Heroku to deploy.
- PostGIS and PostgreSQL as a database.
- Terrain Tiles on AWS as a data source for the initial sculpture.