To the Moon: Creating a Data Architecture Solution for Cryptocurrency Exchange Data Visualization and Analysis

This is a final project for our Data Engineering class held from January to June 2021. This was created alongside Adriane Amorado and Joanna Quinto. For our final project, we were tasked to create an end-to-end data architecture that should have ticked off the following:

1. Input can be via an API endpoint that you will create or by watching a resource e.g., S3
2. Output must be an API endpoint and consumed by a dashboard/visualization or application
3. It must have the following:
   1. OLTP
   2. OLAP
   3. NoSQL database
   4. Data Lake
   5. Automated or periodic ETL/ELT
4. Provide data scientist access
5. Implemented on AWS. However, Redshift clusters should not be kept running and instead snapshots should be stored.

It was a tall order, and we had difficulty thinking of a specific use case that ticked off everything. Nonetheless, we made it work.

Objective

Because a lot of our friends in class were very much invested in cryptocurrency (at the time), we decided to create an end-to-end data architecture that pulls data from Binance and displays the data at the end.

End product

As a proof of concept, we built a dashboard using AWS Quicksight which is connected to our Redshift cluster.

In construction

We are currently collectively writing an article detailing our journey, a more distilled version of our final paper for the course. Stay tuned for updates on that one!