Beautiful data visualizations reveal stories that numbers just cannot simply tell. Using visualizations, we can get a sense of scale, speed, direction, and trend of the data. Additionally, we can draw the attention of the audience – the key to any successful presentation – in a way that’s impossible with tabulations. While a tabular view of new online signups is informative for tracking, a dynamic map would provide a more captivating view and reveal dimensions that a table cannot.
Hence, I worked on a map visualization that depicts signups in real time. In this post we will walk through the tools used to construct this map and discuss the technology that allows the frontend to listen and to receive data from the backend. The code should be sufficient for the readers to build their own flavor of the real-time map visualization. Note that I’m not a front-end developer. I did this for the sake of curiosity!
First off, let’s talk about tooling.
Tools used
- Express.js – backend API server
- Server-Sent Events (SSE) – API that continuously sends data from server to browser
- React.js – frontend user interface
- D3.js – low-level visualization tool
Method
Considering that there won’t be any interaction from the viewer that will touch the server, employing an older technology called Server-Sent Events (SSE) would make it easy for the Express backend to send new data to the browser without requiring the client to make the initial request. The tradeoff of using SSE is that it lacks the ability to handle data sent from the client. In other words, SSEs establish a unidirectional channel flow between the server and the client. Since this is a view-only application, this solution works just fine. In the browser, SSEs appear via the EventSource interface, which listens to the server for new events, and has a callback that fires whenever an event appears.1
Setting up the Express.js API server
The JavaScript pseudocode below writes an endpoint in the Express server API to which the client can listen for new data. Note that connect and ping are two callback functions: one establishes a connection to the data source containing information on latitude and longitude of each event, and the other pings the data source for new data, respectively.
// Express.js API server
const express = require('express');
const app = express();
function connect(callback) {
// connect to a data source
};
function ping(callback) {
// ping for data from the source
};
app.get('/update-stream', (req, res) => {
// create the response headers
res.writeHead(200, {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive'
});
// connect to the data source
connect(() => {
setInterval(() => {
// ping the data source
ping((data) => {
// each response must have a unique id!
res.write('id: ${i++}\n');
// convert the result to a JSON string and write to the client
res.write('data: ${JSON.stringify(data)}\n\n');
});
// ping every two seconds
}, 2000);
});
});
let server = app.listen(3000, () => {
let port = server.address().port
console.log('Map listening at http://localhost:%s/', port)
});
The backend executes setInterval to ping the data source every x seconds. The results are emitted to the client.
Receiving the data with React.js
The frontend continuously listens to the channel stream, and any new data is then passed into a React component via the EventSource by the Express backend. In the React component:
// upon mounting, begin listening to update-stream endpoint
componentDidMount() {
// opens a connection to the server
const source = new EventSource('update-stream');
source.addEventListener('message', this.handleData.bind(this));
privates.set(this, source);
};
where handleData is a callback that handles the data and privates is a WeakMap object.
Rendering with D3.js
Finally, the data gets passed into the D3 component, which is responsible for converting the latitude and longitude to their respective x and y coordinate points in the browser, as described in this block. Voilà !
Conclusion
The process is simple. The backend periodically pings the data source for new data. The data is then emitted to the client via the EventSource continuous server connection. At this point, the data gets passed into a D3 component that updates the visualization with the new points.
I’m happy with how it turned out and definitely plan to look into some more real-time visualization opportunities.
1 Now, I bet you’re thinking why not use WebSockets? I wanted simplicity and found that this approach was the least complicated way to retrieve and continuously stream new data. Each ping takes less than a second to complete, so there was no risk of requests queueing up. ←
