Log in
Subscribe
Aug 23, 2021
Building a Contact Tracing Platform
Building a Contact Tracing Platform
00:00
39:38
Transcript
0:00
[on hold music] Infrastructure as code is the best way to manage the deployment and maintenance of applications as well as the hardware required to operate them.
0:19
Writing configurations for applications as code allows for easy, repeatable, and identical deployments across multiple environments.
0:27
In this article, I'm gonna describe to you how to use the AWS Cloud Development Kit, CDK, to create an application and deploy it to an AWS environment.
0:37
AWS CDK is an open source framework for defining cloud application resources, including compute, storage, and security resources. It's available in a number of languages, including TypeScript, Golang,.NET, and Java.
0:52
The advantage to using CDK is that you can develop the resources for your application in a language your business already uses, so developers don't need to learn a new language to deploy and configure an application.
1:04
To show you how easy it is to create an application with CDK, I'm gonna make something topical. At the time of writing, Melbourne is under its sixth COVID lockdown.
1:13
One of the reasons given for the lockdown was that the contact tracing personnel were inundated with exposure sites which needed to be notified, which then needed their staff and customers to be isolated and tested while their whereabouts were traced.
1:27
This type of problem can be automated, so I figured, why don't I try and build a contact tracing app?
1:32
If you want to see the entire project or a particular snippet of the code I've written for this demo, you can find it on GitHub. Architecture backend.
1:40
To start with, here's the basic architecture diagram of what the backend service looks like. The backend architecture for this application is quite simple.
1:48
The system is based off of a serverless architecture, so its costs is proportional to the amount of use it receives.
1:55
The API, which will handle the contact tracing, is managed through Amazon AppSync, a managed GraphQL service. GraphQL is a language and specification for data query and manipulation used most commonly with APIs.
2:08
AppSync is a great service because it gives front-end developers the ability to query multiple databases, microservices, and APIs with a single endpoint.
2:18
Since there's only one endpoint, it's really easy to update the API in real time by just adding more data sources.
2:24
To create an API, all you need to do is define a schema and where to fetch the data from, and mostly you're done.
2:31
There are a few more steps if you want to define authentication and more complicated data sources, but for the most part, it's plug and play. The check-in data for this application will be stored in Amazon DynamoDB.
2:44
There are a few reasons why I chose Dynamo. The first is that it's a fully managed serverless product, so I don't have to worry about managing a database on a server. The second is that it's fast, and I mean really fast.
2:56
A standard query takes about five milliseconds to complete. That's pretty fast.
3:01
As we'll go on, I'll talk about how I'm caching a lot of the queries in the Dynamo accelerator, but this didn't speed up my queries for the most part, and I'll explain why later.
3:11
The final reason is that the type of data being used can be easily stored in Dynamo as key value pairs. Each check-in looks something like this example.
3:22
Super nice and simple, and it's really easy to search for a check-in based on the time, user ID, and location ID.
3:29
DynamoDB queries need a partition key to search upon, a primary key analog, and can be refined using a sort key.
3:37
In the application, I'm using the user ID and location ID as pa-partition keys, while the date will be the sort key. DynamoDB tables support a single partition key.
3:48
To use multiple partition keys in DynamoDB, you need to use a global secondary index. Authentication is vital for real world services.
3:57
Having the ability to verify users who will access sensitive data, check-in data is personal identifiable information or PII, is absolutely needed.
4:07
AppSync supports multiple simultaneous authentication methods, including API keys, IAM user signatures, Amazon Cognito, and external OpenID providers.
4:17
For this demo, I'm gonna use Amazon Cognito, which is a managed authentication service that can store thousands of user credentials securely in the cloud. There are two components for the API in the lib directory.
4:29
The functions directory contains all the API code, and the GraphQL directory contains the schemas. What is CDK? The AWS CDK is a framework that can be used to define and organize resources on the AWS cloud.
4:46
Any project in CDK is set up using stacks, which are individual deployments that resources are grouped by.
4:53
In actual fact, CDK stacks are synthesized into Amazon CloudFormation templates and deployed using CloudFormation.
4:59
CDK is available in a bunch of different languages, but for this demo, I'm gonna use the TypeScript one to define my CDK stack. I'm using CDK version two in this demo because it's pretty close to being released soon.
5:11
CDK V2 has a bunch of improvements over the first version, which you can read about on this AWS blog post. But for this demo, it won't matter too much which version you use.
5:21
The first thing to do is to install the CDK CLI. Now, if you're using another language like Go or Java, the installation will be a bit different, but there are instructions for each language on the CDK website.
5:33
For Node, all you have to do is use NPM install -g AWS-CDK. After this, you can create a new CDK project in an empty directory based off the app template as shown below by running CDK init app, language TypeScript.
5:50
After th... After this, you should have a blank CDK project. Woo. To verify, you should have a directory layout similar to the output below. Defining the schema.GraphQL is the first and foremost a query language.
6:06
It's used to describe the data in an API, ask for specific data, and return predictable results. All queries to a GraphQL API are sent to one HTTP endpoint which supports POST requests.
6:20
The data sent in the POST request defines the query or mutation and what information to return. For the GraphQL engine and clients to understand what queries are valid, they require a schema.
6:32
The GraphQL schema is a great tool for defining the scope of the API. It has a lot of advantages over traditional HTTP endpoints schemas. The first is that it's self-documenting.
6:43
Since the API is defined by the schema, it's always exactly what queries the GraphQL engine is expecting.
6:50
Secondly, any GraphQL inspector like GraphiQL can automatically read the schema and lint your requests as you write them once you've set up the authentication with the server.
7:01
Lastly, since GraphQL is strictly typed language, all the requests defined in the schema have types that can be used, expanded, and checked against, meaning it's very hard to send an incorrect query.
7:13
For this project, there are two query files, and the first is the scalas.graphql file. This is used for linting purposes during development.
7:22
The GraphQL specification defines a few specific data types that all engines must support, things like String and Float, but any engine implementation can have extra built-in types.
7:32
In fact, you can specify your own in the schema. AppSync has several built-in types that we're going to use. So help...
7:39
so to help tools like VS Code and IntelliJ understand these new types, we can define another schema file that is in the same directory as our main schema that won't get used by the CDK stack but will be used by any tool that scans the directory.
7:54
For a complete list of AppSync built-in scalar types, you can check out the documentation on the site. Writing the functions. The backbone of an AppSync API are the data sources.
8:08
The data sources map to a query or mutation object and tell AppSync where to get each set of data for the operation.
8:16
At the time of writing, there are six different data sources: DynamoDB, Elasticsearch, Lambda, RDS, HTTP, and None, that's just an empty placeholder. This API uses two different data sources.
8:29
For simple key retrievals, the API has a DynamoDB resolver. This data source can query the table directly using a mapping template and return the results. There are a few advantages to doing this.
8:40
The main one is that you don't need to write an external Lambda function in another language to query the database. The other resolver used is the Lambda provider.
8:49
AppSync can use Amazon Lambda functions as a data source provider. Using Lambda to run functions allows your API to run any arbitrary code needed to get a result for the API.
9:00
Also, since Lambda functions can now run Docker images, there isn't really any function that your API can't do.
9:08
To set up a Go project inside the CDK, the only step you need to complete is creating a Go manifest file using Go Mod. The manifest file is used by the Go toolchain to track and download dependencies for a program.
9:23
It's also used in the build steps to validate the structure and version of Go the program is compatible with. After ma...
9:31
after making the manifest file, you can create a Go file in any subdirectory, which is a great way to organize the code that you're going to write.
9:40
Below is an example of how I've laid out the structure of the contact tracing API functions. A quick note on CDK v2 and Golang Lambda functions.
9:51
At the time of writing this article, one of the major flaws of CDK version two is that there is no support for using languages other than JavaScript to write Lambda functions with and have them automatically bum- bundled and deployed with the stack.
10:05
For the first version of the SDK, Rafael Wolinski wrote a custom construct that will compile, compile Golang Lambda functions and bundle them with the stack. However, this con- construct doesn't work with v2 of the CDK.
10:19
So to fix this, I spent the better part of a weekend a few weeks ago patching up the code that Rafael wrote and making a version that's compatible with version two of the CDK, which you can find on GitHub. Common.
10:33
A lot of code to access the Dynamo table across functions is identical, so I've refactored this code into its own separate file.
10:40
The utilities.go file has two public methods: GetLocationVisitors and GetUserLocationHistory.
10:48
In the snippet below, I've only added the code required for GetLocationVisitors as an example of how to build a function that can access Dynamo.
10:58
There are a few components from the above code I'd like to highlight before moving on.
11:02
The first is that the check instruct is using DynamoDB AV tags to define the relationship between struct properties and DynamoDB properties. Tags are using Golang to provide additional context to properties in structs.
11:19
They're most commonly used for unmarshaling or marshaling JSON and XML data with structs without the need for custom code to handle the process. The second point is that the code around DAX.
11:31
In this app, I'm using DynamoDB Accelerator, or DAX, to cache the results of queries and speed up the API response time.
11:39
I'll go into more details about setting up and configuring DAX later, but for the time being, whenever you see DAX code in the API, you should know that you can also use DynamoDB clients as well.
11:50
The final point to mention revolves around the pagination of results received from DynamoDB. Any query that is made to DynamoDB will return a maximum of one megabyte tr- per transaction.
12:02
To collect all the results from a query, you need to use the paginator object.The new query paginator will handle the work collecting multiple pages of data and is a drop-in replacement for the standard query method.
12:16
Trace exposure function. With the Dynamo table access separated into different files, the contact tracing code can import the calls when needed using the import statement.
12:28
The structure of the code is fairly straightforward, so I won't get into too much detail. The flowchart below describes how the algorithm works.
12:38
At this point, I'd like to mention that the Lambda resolver that I've set up is known more specifically as a direct Lambda resolver.
12:46
The distinction is because when using a Lambda resolver, you can optionally include a mapping template that can transform the API input before the Lambda sees it.
12:54
To understand more about how direct Lambda resolvers work, you can read this blog about them.
13:00
Because there is no mapping template for the function, you need to take extra care in parsing and validating the input of the function.
13:07
Because Lambda doesn't have any information about how t-the input event is structured, it is defined using an empty interface, which is the Go analog of an any object.
13:18
Because of this, you have to test and type assert any object that should be inside of the input. Building the stack. Now that the function code has been defined, we can move on to building the stack definition.
13:31
Inside of the lib directory, there should be a file similar to contact-tracing-stack.ts. This file contains the stack and resources that will be added to the app.
13:41
Inside the file, there should be a single class which will be the stack definition. From here, all the resources will be defined within the scope of the class constructor. VPC. It's required to set up a DAX server.
13:55
VPCs are split into subnets which can be connected to the internet, private versus public, and given unique names and subnet masks. DynamoDB table. There are two resources that need to be created for the Dynamo database.
14:11
The first is the table itself, and the second is a global secondary index. The Dynamo table will allow querying using a single partition key and optional sort key for this table.
14:22
The location ID will be the partition key and the check-in date will be the sort key. This will mean that queries the table based by location will be trivial, but it won't be possible to run a query based on the user.
14:36
The way to solve this is to use DynamoDB to create another index. There are two types of secondary indexes in Dynamo.
14:44
Global secondary indexes allow for another partition key, while local secondary indexes provide another sort key.
14:51
Since we're going to be searching for a user u-user and sorting the results based on the check-in date, we only need a global secondary index. Using Dynamo DAX.
15:03
Over the course of this project, I found that some of the larger requests were taking up to sixteen seconds to run. In terms of making a responsive website, that's incredibly slow.
15:12
As I investigated the issue, I found that over ninety-nine percent of the execution time was being used writing for DynamoDB queries. Now, like I said at the top of this article, Dynamo is really fast.
15:23
Queries usually take less than five milliseconds to run. But when you're creating a contact tracer that has to cross-reference thousands, potentially millions of people and locations, those five milliseconds can add up.
15:35
You can see the latency in each segment represents a query operation, as you can see the number of queries rapidly increasing.
15:43
To speed up these requests, I decided to use DynamoDB Accelerator, DAX, to cache the results of any Dynamo queries.
15:50
DAX is an in-memory cache that can deliver up to a ten times performance improvement for DynamoDB queries. It does this by creating a cluster in your VPC that can communicate with any application connected in that VPC.
16:04
DAX is a drop-in replacement for DynamoDB, meaning that if you write code to work with DynamoDB, all you need to do is replace the DynamoDB client with the DAX client and it will work right away.
16:14
There's no need to change any of the application logic to support DAX API calls. My initial thought was that most of the queries that would be executed would be repeats of previous queries.
16:26
This as-assumption, however, was wrong. I'd like to go on a little rant now. So as I'll mention earlier in this article, I'm going to use Golang to write the Lambda functions.
16:37
To use the AWS API in Golang, you should use the official SDKs.
16:42
The latest version of the SDK, AWS SDK version two, has been in active public development for a few years and was classified generally available in January of this year.
16:53
Now, the new SDK has a bunch of useful features that make it easier to use, which I've mentioned in the highlights. You can see a full list of improvements on the SDK website.
17:03
But for the time, but for the time being, I wrote most of the functional code before adding support for DAX. Now, the new SDK does not have out-of-the-box support for DAX.
17:14
Due to the architecture of how DAX works, it would require a major rewrite. The original SDK doesn't have support either, but there is another SDK built for DAX that has support for version one of the Golang SDK.
17:26
Now herein lies the problem. Because version two of the Golang SDK is a complete rewrite, this library is incompatible with the new SDK version.
17:34
Now, you'd think that there would be support for the new SDK considering both of these repos are the official AWS repositories, but alas.
17:42
So I took it upon myself to try and add support for the new version, which you can see on GitHub. So sounds great, like the problem is solved, right? Well, no, because the DAX protocol is pretty confusing.
17:52
To get the speed they're promising, the DAX team isn't using HTTP to talk to the DAX cluster. They're rolling their own custom protocol using TCP.
18:00
So all I've done is change the input and output structs of the DAX SDK to support the structs from version two of the AWS Golang SDK. Is it an elegant solution? No. Is a good solution? Eh, but it is a solution.
18:15
Rant over.So anyway, if you wanna use DAX with this application or any other AWS SDK version two code, you need to add this snippet to the bottom of your go.mod file to let the toolkit know that you want to use a fork of the original repository.
18:31
After all this work, I tested the API calls and success, kind of. Well, the first request is still slow, but any identical subsequent requests are much faster, less than one millisecond.
18:43
The only problem w-with this is that most of the requests are not gonna be identical, so the speed up isn't really worth the effort.
18:50
I've documented what I've done for the sake of completeness, but I don't recommend using DAX for a problem like this. Cognito user pool and client.
18:59
Next, to set up Cognito, there are two resources that need to be created. The first is a Cognito user pool. This resource will hold all the information about users who access the API, mainly their email and password.
19:11
The second component is the Cognito app client. This resource is used as an endpoint for web services to talk to Cognito. It specifies what authentication methods are valid. AppSync API schema.
19:23
Now with the dependencies out of the way, we can now create the AppSync API. The first part is to define a new AppSync API resource and the schema the resource will use.
19:33
This will create an AppSync API that will use the attached scream-- schema that we created before. But at the moment, none of the data values in the schema are connected to a resolver.
19:43
To do this, we'll need to create some data sources and resolvers. AppSync data sources and resolvers. The data sources and resolvers are a critical piece of the API infrastructure.
19:55
They connect AppSync to all the storage and compute resources behind your API, so making sure they're correctly defined is crucial. The first data source we're going to make is for a direct DynamoDB connection.
20:08
To make this, we're gonna give AppSync full access to our DynamoDB to read and write to our table. We're then going to point AppSync to the DynamoDB table we created before.
20:18
Now we need to make a resolver for our data source.
20:21
This resolver will be linked to the get user location history and get location attendees API queries, which will be used by the API to get results for the specific users and locations on the pages ct.vo cel.app/location and ct.vo cel.app.
20:37
Direct DynamoDB resolvers are useful when you don't need to use much business logic to return API data. These resolvers only need a mapping template that can be used to convert the input into a DynamoDB transaction.
20:50
The result is then converted into a JSON compatible string and returned to AppSync. The mapping template is written in a language called Apache Velocity.
21:01
You can find more about it in the AWS documentation, including a list of helper functions for working with DynamoDB. We're also gonna use this data source to put new check-ins into the table.
21:15
Infrastructure as code is the best way to manage the deployment and maintenance of applications as well as the hardware required to operate them.
21:23
Running configurations to applications as code allows for easy, repeatable, and identical deployments across multiple environments. In this article, I'm gonna describe how to use the AWS Cloud Development Kit, CDK,
21:36
to create an application and deploy it to an AWS environment. AWS CDK is an open source framework for defining cloud application resources, including compute, storage, and security resources.
21:49
It's available in a number of languages including TypeScript, Go Lang,.NET, and Java.
21:55
The advantage to using CDK is that you can develop the resources for your application in a language your business already uses, so developers don't need to learn a new language to deploy and configure an application.
22:07
To show you how easy it is to create an application with CDK, I am going to make something topical. At the time of writing, Melbourne is under its sixth COVID-related lockdown.
22:17
One of the reasons given for the lockdowns was that the contact tracing personnel were inundated with exposure sites which needed to be notified, which then needed their staff and customers to be isolated and tested while their whereabouts were traced.
22:31
This type of problem can be automated, so I figured, why don't I try and build a contact tracing app?
22:37
If you wanna see the entire project or a particular snippet of the code I've written for this demo, you can see and find it on GitHub.
22:50
This process is exactly the same as the above query, but instead it will use the put item operation. That's it. Now, get user location history, get location attendees, and check in are connected to the data source.
23:05
The final call we'll look at is trace exposure flat. First, we're gonna create a Go Lang function using the plugin that I mentioned in a quick note on CDK version two and Go Lang Lambda functions.
23:17
And we're gonna set some environment variables from resources we've already created and create some security policies to grant access to the resources.
23:26
Now we're gonna create a service role that AppSync can use to invoke the Lambda function, as well as create a Lambda data source.
23:34
After we've created a data source, we're going to connect it to a resolver like we did before. And there you have it. Now the Lambda function we wrote before is connected to AppSync and can be used for API calls.
23:47
All that's left to do is to deploy the stack. Deploying with CDK. Deploying with CDK is easy to do.
23:55
There is a single command that will generate the CloudFormation template and deploy it to the configured environment, CDK deploy.
24:02
One thing to mention here, a useful feature of CDK is the ability to use CloudFormation outputs to print attributes that are useful.
24:10
For example, the API URL, user pool ID, and app client ID are all values needed by the front end, which can be hard-coded into the environmentFront end. The front end of the application is a React site using Next.js.
24:29
Next is a great framework for developing websites using React, and can be hosted on any static web service.
24:34
For this application, I'm using Vercel.com, as it's a free service that works really well with Next sites since they're both made by the same team.
24:43
I do wanna point out that Amazon Amplify has great support for Next.js as well, but I'm using Vercel as my preferred choice. The first step is to create a Next app.
24:52
This can be done with the Create Next app tool, which is a pretty similar to Create React app, if you're familiar with that. NPX Create Next app front end.
25:04
After running this setup command, you should have a directory, something like this. We now have a boilerplate Next.js project running NPM run dev.
25:12
We'll start up that deployment server, and you should see a starting page. We're not gonna be using this page or the API setup by Next, so you can delete the pages directory.
25:23
We're gonna make two new files in the sources page directory, underscore app dot TSX and index dot TSX. They should look something like this.
25:33
After creating these pages, if you run NPM run dev, they will warn you that you don't have TypeScript or the right types installed. To fix this, run the following install script, NPM I save dev TypeScript types React.
25:49
Now, when you start the dev environment, you should get a blank Hello World page. Now we can set up the components of the app. Setting up authentication. When we set the back end...
26:01
When we set up the back end, we created a Cognito user pool and client, and connected our user pool to AppSync. We're now going to use our client to get an authentication token that will work with AppSync.
26:13
This flow should work something like this. Number one, the user loads the website. Number two, the website checks to see if the user is logged in. Number three, if yes, go to step five.
26:24
Number four, direct the user to the login or sign up page. Number five, when the user logged in, ask Cognito for an authorization token to send with API requests. Number six, Cognito returns a token that can be used.
26:37
These steps are pretty straightforward, but can be very tricky to do correctly. Fortunately, there's a pre-built library for Cognito authentication that we can use in the AWS Amplify package.
26:47
NPM install save dev AWS dash Amplify. Once the package is installed, we can configure the auth object to connect to our Cognito client.
26:57
For this, we need the ID of the user pool and the client which we have exported from our stack before. Let's save them for the time being and add them to the code as environment variables.
27:09
Connecting authenticator to React. To get an authentication token, you need to build a login flow for the app. Now, there are two ways this can be done.
27:19
You can either use the pre-built Amplify login screen, or you can roll your own login. I've decided to build my own login flow, which I made for another project.
27:27
If you wanna see the relevant pages, they're available in the repo. To make sure that the page shown, i- to the user is always authenticated, I've made a simple hook to return the authentication state to the page.
27:39
Be aware this is a simple example of what should really be built into re- a React provider, but it's fine for the example purposes. Setting up Apollo.
27:50
Now that our authentication is set up, we can connect the AppSync API, and run queries and mutations. Apollo is a GraphQL implementation that can be used to generate and create an API similar to AppSync.
28:02
For our use, however, we're gonna be using the open source client library that Apollo provides to execute queries against our endpoint. First thing we need to do is install Apollo. NPM install save dev Apollo Client.
28:16
After Apollo is installed, we need to configure it to work with a Next.js React app and the authentication methods we've built. There is an example in the Next.js repo that explains how to connect Apollo.
28:29
To connect with the Amplify auth library, all we need to do is fetch the token, which can be seen in the auth link context.
28:38
The Apollo Client creates a React hook that can be used by Next as a provider, a type of React component that provides a context.
28:48
A context provides a way to pass data through the component tree without having to pass props down manually at every level.
28:55
Now we can wrap our app in the Apollo Provider, making our API available to any React component in the tree. Now that we build the data components in our app, we can automatically get data from our API.
29:10
Getting data from the API to our page. Now that we have our authentication and data layer built, let's test them by displaying some content on our index dot TSX page.
29:23
We're going to make a simple table to s- to show the data for a single user. To start with, we'll define the query we want to execute on the page.
29:31
We'll call it Get User Location History, and save it in a file that we can import from later. Then we can build a simple React page that can display the user table component if the user is authenticated.
29:45
The user table component is where all the business logic is found. We're going to use the use lazy query hook to get the data from our API when a user presses a button on the page.
29:57
Now, when you reload the page, you should see something like this. Building D3 components. So now that we have the data available to React, we can now do something with it.
30:11
Since the user is on a site to view contact tracing data, they probably wanna see the data they requested.
30:17
There's a lot of different ways to view this data, but I'm going to explain how I made two of the layouts in the site in the article, otherwise this post would never get finished.
30:27
We're going to make a radial tree and a force directed tree.There are lots of ways to display data on a website. The earliest and most mundane system is to simply use tables like we did before.
30:39
But using tables to display data can often obscure the more intricate connections and details in the data.
30:45
Having custom elements that can display the data in different ways to emphasize and highlight specific facts and connections is a much more user-focused approach.
30:54
To help with this, we're going to be using a library called D3, a JavaScript library for imbe-- mm, a JavaScript library for manipulating documents based on data.
31:05
D3 allows you to bind arbitrary data to the Document Object Model, or DOM, and then apply data-driven transformations to a document.
31:13
This approach is incredibly powerful, allowing developers to build anything within the scope of the DOM's APIs. Another advantage of D3 is the large set of utility modules that come included.
31:25
While building the different components, we're going to use a fair few of them. Radial tree. The first component we're going to make is a radial tree.
31:34
This is a type of tree that has all the nodes sorted and aligned around a central root node. It's really useful for understanding the depth of a tree and how sparse or dense it is.
31:43
To start off, we're gonna make an empty Next.js page, which will render an SVG component. This should now render a blank page when you navigate to /radial-tree.
31:53
Now it's time to add our connection to the AppSync API using Apollo. The Apollo client library for React manages both local and remote data with GraphQL.
32:05
To get the data from the server onto our page, we're going to use the useLazyQuery React hook. This method will take the query string and variables and send these to the server. First, we'll define the query to execute.
32:20
We can place this in a separate file and export it into the component. This will be useful later as multiple components need to use the same query, plus it's also tidier.
32:31
I'm using a custom component I made called SearchBox, which just creates a form and validates the inputs using Formik.
32:38
You can read more about it in the repo, but for the time being, assume it's a form and the on submit function runs when the form is submitted.
32:50
So now when the user selects a date and user ID, the API will be called through the React layer and return data to the view. Now all that needs to be done is display it.
33:02
There are a lot of moving parts in modern React applications. Let's just pause for a moment and understand what we're doing and what the tools and frameworks we're using are designed for.
33:12
First, React is a library for building user interfaces. In the traditional MVC design pattern, React is the controller component. Its only job is to accept inputs and convert commands for the model or view.
33:25
Apollo is a framework for data and state management. It's designed to fetch data, keep it in sync with the server, and relay changes. It's the model in the MVC pattern. The final part is the view.
33:38
We have a list of data that needs to be displayed to the user after they make a query.
33:42
As alluded before, to display the data we're going to be using D3, which is a JavaScript library for creating interactive visualizations in the browser.
33:50
It's essentially a way of generating shapes and charts using SVG components and HTML canvases. But the way it does this is amazing.
33:58
There are so many tools and helper functions to build scales, shapes, charts, lines, and colors that you can pretty much create anything you can think of. The advantage to D3 is the ability to create anything.
34:11
The disadvantage is that you have the ability to create anything. There is no pre-built components. You need to know what you want to make.
34:18
To begin with, we're gonna create another React hook that is dependent on the size of the window and the data.
34:24
The hook is going to create an SVG se- selection using D3 that we can use to manipulate and edit the contents of the SVP.
34:33
The hook is going to create an SVG selection using D3 that we can use to manipulate and edit the contents of the SVG page.
34:42
So now that we've got a selectable SVG element, we can put our data into it and produce our visualization.
34:49
To do this, we need to build a data structure that D3 understands and can parse to populate the SVG nodes that it will create. There are a few components that we need to create to do this.
35:00
The first thing that we need to do is clone the data received from Apollo. This is because Apollo data is not extensible.
35:08
We're going to use the stratify constructor cr-- to create an operator to parse our API data into a D3 hierarchy. This is just a predefined data structure that D3 tree functions can understand.
35:21
Finally, we need to create a method that will create a tree from our data. The tree builder is exactly what we need.
35:28
Combining all of these constructor elements together and adding them to our hook gives the following code. Great. So now that we have the methods and structures in place to add data to the page, we're almost there.
35:40
The last thing that needs to be done is to build the tree visually using SVG elements. In the tree, there are two separate sections, vertices and edges.
35:49
Using the D3 select tool, we can create a group of SVG path elements to represent the tree edges.
35:56
The coordinates in the canvas are calculated using the tree builder, so we only need to call the links method to pull the data for each element.
36:05
The vertices can be created in a similar fashion by building circle SVG elements and calling the descendants method to retrieve all the vertex data. The entire SVG generation can be seen in the below snippet.
36:19
There is some code that explains how the tree is wrapped into a circle as well.
36:26
After adding all these components, if you refresh the page, you should see something like the tree below.If everything is working, you can now generate a tree that shows the contacts and locations exposed to a person, but it's pretty hard to gleam any usable information from this tree at the moment.
36:42
We can fix this by adding some contextual data using event listeners and add vertex and edge information to the tree when a particular element is hovered over.
36:54
Now, when you hover over part of the tree, you should see a handy tooltip. Force-directed tree.
37:02
Another visual we can create is the force-directed tree, a visualization that uses a numerical integrator for simulating forces. This visualization is really good at showing clusters and relative sizes of tree groups.
37:16
The setup and implementation is identical to the radial tree we created before. The only difference is the D3 render hook. In this, we're going to create a simulation object and link it to our tree structure.
37:29
If you create a new file and copy the code from radialtree.tsx and replace the hook with the code seen above, you should see something like this. Mapping using Mapbox.
37:43
Visual components are great, but sometimes you need to display the data relative to the real world. Maps are the best way of displaying geographical data.
37:51
In this component, I'll show you how to overlay geographical data onto a map. We're going to make another page called map.tsx, which will be our base. To make the map, we're going to use a library called Mapbox.
38:03
Mapbox is a great service that provides high-resolution maps with their SDK. To set up map, we're going to import the JavaScript SDK and the map styles. We're also going to create a div, which will be the map container.
38:16
After creating the page, we need to initialize the map inside the container. We're going to use the useCallback hook to make the custom references for our map that can be updated when the page loads the SDK.
38:27
When the map is initialized, we can begin displaying data using a React hook similar to how we display data in the D3 components. And there you have it. Now the map can load data and display it on a page.
38:40
If you refresh the page, you should see something like the map below. Some final thoughts. If you've made it this far, well done.
38:47
This article turned out to be a lot longer than I originally planned, and I thought about splitting it into multiple articles, but I think the flow of a single document is easy to understand and consume.
38:57
This project was actually a lot more complicated than I first assumed it would be. The fact that both DynamoDAX and X-Ray didn't have support for Golang's AWS SDK version 2 really slowed down my development.
39:08
The D3 charts also took a long time to make, but once they started to work, it was okay to iterate and improve.
39:15
If you found this article useful or can't understand a word of what I've said, you can yell at me on Twitter. My handle is Koshi.
Kochie Engineering
Listen on
Apple Podcasts
Apple Podcasts
Spotify
Spotify
YouTube
YouTube
Pocket Casts
Pocket Casts
Recent episodes
Halo Physics
May 28, 2021
