Using Portal’s Datasets

The Filecoin Data Portal publishes up to date dataset on a daily bases as static Parquet files. You can then use any tool you want to explore and use these datasets! Let’s go through some examples.

Python

You can use the pandas library to read the Parquet files. Here’s an example:

import pandas as pd

url = 'https://data.filecoindataportal.xyz/filecoin_daily_metrics.parquet'
df = pd.read_parquet(url)
print(df)

You can play with the datasets in Google Colab for free. Check this sample notebook.

JavaScript

You can use the duckdb Obervable client library to read the Parquet files and run SQL queries on them. Here’s an example:

db = DuckDBClient.of({})

db.query(`
  select
    *
  from "https://data.filecoindataportal.xyz/filecoin_daily_metrics.parquet"
`)

Check this sample Observable JS notebook to see how to explore and visualize the datasets.

Dune

Some of the datasets built by the pipelines are also available in Dune. You can use the Dune SQL editor to run queries on these datasets. Here’s an example:

select
  date,
  onboarded_data_pibs,
  unique_data_onboarded_data_pibs,
  data_on_active_deals_pibs,
  unique_data_on_active_deals_pibs,
  deals
from dune.kalen.dataset_filecoin_daily_metrics

Google Sheets

The pipelines that are executed to generate the datasets are also pushing the data to Google Sheets. You can access the data directly from these Google Sheets:

• Filecoin Daily Metrics
• Filecoin Storage Providers
• Filecoin Clients
• Filecoin Allocators

You can create a new personal Google Sheet and use the IMPORTRANGE function to read data from these sheets and be able to plot or add more transformations on top.

BI Tools

Depending on the BI tool you are using, you can connect to the Parquet files directly, use the Google Sheets as a data source, or you’ll need to load the data into a database like PostgreSQL or BigQuery. There are

Evidence

Filecoin Pulse is a website build with Evidence using the Filecoin Data Portal datasets. You can check the source code on GitHub to see how to use the datasets in Evidence.

Observable Framework

Another alternative is to use the Observable framework to create dashboards and visualizations. You can use parquet files as data sources and generate beautiful static websites providing dashboards and reports like Filecoin Metrics, a proof of concept dashboard built with Observable Framework. You can check the source code on GitHub too.

Others

Do you have any other tool you want to use to explore the datasets? Reach out and let’s explore how to use the datasets with your favorite tools!