A Comprehensive Guide to Running a Graph Protocol Indexer and Subgraph for Cosmos Chains

Note

In our earlier blog, we discussed how to index a Cosmos chain using the Graph protocol. This follow-up post dives deeper into specific aspects to provide a more comprehensive understanding, enabling you to effectively manage your self-hosted indexer and subgraph.

In the ever-evolving landscape of blockchain technology, efficient data indexing and querying are paramount. The Graph Protocol provides a decentralized and robust solution to index and query data from blockchains. By leveraging The Graph, developers can build and query open APIs, known as subgraphs, making blockchain data easily accessible and searchable. This capability is essential for creating dynamic, data-driven applications and services. We rely on it to efficiently load Gitopia’s user dashboard.

In our previous blog post, we explored how to integrate The Graph Protocol with the Cosmos blockchain. Today, we will delve deeper into the implementation details and provide a comprehensive setup guide for running an indexer. This guide will help you maintain continuous indexing, even as the gitopiad binary evolves through various upgrades. This setup is crucial for Gitopia and serves as a template for other Cosmos chains aiming to implement similar solutions.

Prerequisites

Firehose-Cosmos: Required to process and stream blockchain data efficiently.
All versions of gitopiad binaries and their respective upgrade heights.
Node.js, Docker, Git: Essential development tools.
A JSON or YAML configuration file to track binary versions and upgrade heights.

Setting Up the Development Environment

Ensure your environment is ready by installing necessary tools and configuring Firehose-Cosmos.

Step 1: Install Required Tools

Install Go

#!/bin/bash
wget https://golang.org/dl/go1.19.9.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.19.9.linux-amd64.tar.gz
sudo sh -c "echo export PATH=$PATH:/usr/local/go/bin >> /etc/profile"
export PATH=$PATH:/usr/local/go/bin

Install Docker

#!/bin/bash
sudo apt-get update -y
sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release \
    build-essential \
    pkg-config \
    make \
    jq

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

echo \
    "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
sudo usermod -aG docker ${USER}
sudo apt-get install -y docker-compose

Install node and yarn

curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - &&\
sudo apt-get install -y nodejs
sudo npm install -g yarn

Setup Filesystem

First you need to check the available disks using lsblk and identify the newly attached disk (usually /dev/sdb) before formatting and mounting it.

Warning

Proceed with caution. Improper formatting or mounting of disks can result in data loss. Please, ensure you know what you’re doing before proceeding.

#!/bin/bash
# This command wipes the disk, make sure you backup important data, if any
sudo mkfs.ext4 -m 0 -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/sdb
sudo mkdir /var/gitopia
sudo mount -o discard,defaults /dev/sdb /var/gitopia
sudo chmod a+x /var/gitopia
sudo chmod a+w /var/gitopia
mkdir -p /var/gitopia/data
sudo chown ubuntu:ubuntu /var/gitopia/lost+found # fixes init errors

Step 2: Prepare `gitopiad` binaries and configuration file

Setup git remote helper

In this section, we will guide you through the process of installing Gitopia’s git remote helper. This helper is essential for git to fetch repositories hosted on Gitopia.

curl https://get.gitopia.com | sudo bash

Setup `gitopiad` binaries for firehose

To add firehose compatibility you just need to replace cometbft or tendermint in go.mod with the graphprotocol fork of tendermint of your version. Let’s start with the latest version to show how to make this change and then follow this for all other binaries for the upgrade heights.

Warning

Make sure to use the correct graphprotocol fork of tendermint for your binaries. We are using the same version for all the binaries since there was no upgrade in tendermint module in any of the binaries.

mkdir -p /var/gitopia/binaries
git clone gitopia://gitopia/gitopia
cd gitopia
git checkout v3.3.0

Now open go.mod and replace this line:

replace github.com/tendermint/tendermint => github.com/cometbft/cometbft v0.34.28

with:

replace github.com/tendermint/tendermint => github.com/graphprotocol/tendermint v0.34.28-fh

go mod tidy
make build
mv build/gitopiad /var/gitopia/binaries/gitopiad_fh_v3.3.0

Repeat the process for other required versions:

git checkout v3.2.0
make build
mv build/gitopiad /var/gitopia/binaries/gitopiad_fh_v3.2.0


git checkout v3.0.0
make build
mv build/gitopiad /var/gitopia/binaries/gitopiad_fh_v3.0.0

git stash # go.mod changes are already done in v2.1.1-improved-events-fh.1
git checkout v2.1.1-improved-events-fh.1
go mod tidy
make build
sudo mv build/gitopiad /usr/local/bin/gitopiad

Create configuration file

Create /var/gitopia/upgrade_config.json to keep track of binary versions and upgrade heights. You can refer your blockchain’s software upgrade proposals to get the respective heights and binary version.

{
  "upgrades": [
    {"height": 6072000, "binary": "/var/gitopia/binaries/gitopiad_fh_v3.0.0"},
    {"height": 6446000, "binary": "/var/gitopia/binaries/gitopiad_fh_v3.2.0"},
    {"height": 6720000, "binary": "/var/gitopia/binaries/gitopiad_fh_v3.3.0"}
  ]
}

Step 3: Automate Firehose Restart

Create a /var/gitopia/firehose.sh to handle stopping Firehose, replacing the binary, and restarting Firehose.

#!/bin/bash

CONFIG_FILE="/var/gitopia/upgrade_config.json"
FIREHOSE_SERVICE="firehose"
JQ_PATH="/usr/bin/jq"
LAST_UPGRADED_HEIGHT_FILE="/var/gitopia/last_upgraded_height"

stop_firehose() {
  echo "Stopping Firehose service..."
  sudo systemctl stop $FIREHOSE_SERVICE
  if [ $? -ne 0 ]; then
    echo "Failed to stop Firehose service"
    exit 1
  fi
}

start_firehose() {
  echo "Starting Firehose service..."
  sudo systemctl start $FIREHOSE_SERVICE
  if [ $? -ne 0 ]; then
    echo "Failed to start Firehose service"
    exit 1
  fi
}

update_binary() {
  local binary_path=$1
  echo "Updating gitopiad binary to $binary_path"
  sudo cp $binary_path /usr/local/bin/gitopiad
  if [ $? -ne 0 ]; then
    echo "Failed to copy gitopiad binary"
    exit 1
  fi
  sudo chmod +x /usr/local/bin/gitopiad
  if [ $? -ne 0 ]; then
    echo "Failed to make gitopiad binary executable"
    exit 1
  fi
}

handle_upgrade() {
  local height=$1
  local binary_path=$2
  echo "Handling upgrade at height $height with binary $binary_path"
  stop_firehose
  update_binary $binary_path
  start_firehose
  echo "Firehose restarted with new binary at height $height"
  echo $height > $LAST_UPGRADED_HEIGHT_FILE
}

# Ensure jq is installed
if ! [ -x "$JQ_PATH" ]; then
  echo "jq could not be found at $JQ_PATH. Please install jq to proceed."
  exit 1
fi

# Fetch the node status and debug the output
status=$(curl -s http://localhost:26657/status)
echo "Node status: $status"

current_height=$(echo $status | $JQ_PATH .result.sync_info.latest_block_height | tr -d '"')
if [ -z "$current_height" ]; then
  echo "Failed to retrieve current block height"
  exit 1
else
  echo "Current block height: $current_height"
fi

if [ ! -f "$CONFIG_FILE" ]; then
  echo "Configuration file $CONFIG_FILE does not exist"
  exit 1
fi

# Get the last upgraded height
if [ -f "$LAST_UPGRADED_HEIGHT_FILE" ]; then
  last_upgraded_height=$(cat $LAST_UPGRADED_HEIGHT_FILE)
else
  last_upgraded_height=0
fi

echo "Last upgraded height: $last_upgraded_height"

# Read and process upgrades
$JQ_PATH -c '.upgrades[]' "$CONFIG_FILE" | while read -r upgrade; do
  height=$(echo $upgrade | $JQ_PATH .height)
  binary=$(echo $upgrade | $JQ_PATH -r .binary)
  echo "Checking upgrade: height=$height, binary=$binary"
  echo "Current block height: $current_height"

  if (( current_height >= height && height > last_upgraded_height )); then
    handle_upgrade $height $binary
  else
    echo "No upgrade needed for height $height"
  fi
done

Step 4: Schedule the Script

Give executable permission to the file

sudo chmod +x /var/gitopia/firehose.sh

Schedule the script to run periodically using a cron job.

# Edit crontab
crontab -e

# Add the following line to run the script every 10 minutes
*/10 * * * * /var/gitopia/firehose.sh  2>&1 | logger -t height-checker

You can check the crontab logs by grepping the syslog logs with the tag that we used above:

grep 'height-checker' /var/log/syslog

Step 5: Setting up the firehose indexer

Firehose is responsible for extracting data from blockchain nodes in a highly efficient manner.

Create a /var/gitopia/config.toml.

[deployment]
[[deployment.rule]]
shard = "primary"
indexers = [ "default" ]

[store]
[store.primary]
connection = "postgresql://graph-node:let-me-in@postgres:5432/graph-node"
pool_size = 10

[chains]
ingestor = "block_ingestor_node"

[chains.gitopia]
shard = "primary"
protocol = "cosmos"
provider = [
  { label = "gitopia", details = { type = "firehose", url = "http://host.docker.internal:9030" }},
]

Deploying the Graph node

Create a docker-compose.yml file

version: '3'
services:
  ipfs:
    image: ipfs/go-ipfs:v0.10.0
    ports:
      - '5001:5001'
    volumes:
      - /var/gitopia/data/ipfs:/data/ipfs
  postgres:
    image: postgres
    ports:
      - '5432:5432'
    command:
      [
        "postgres",
        "-cshared_preload_libraries=pg_stat_statements"
      ]
    environment:
      POSTGRES_USER: graph-node
      POSTGRES_PASSWORD: let-me-in
      POSTGRES_DB: graph-node
      PGDATA: "/data/postgres"
      POSTGRES_INITDB_ARGS: "-E UTF8 --locale=C"
    volumes:
      - /var/gitopia/data/postgres:/var/lib/postgresql/data
  graph-node:
    image: graphprotocol/graph-node:v0.31.0
    ports:
      - '8000:8000'
      - '8001:8001'
      - '8020:8020'
      - '8030:8030'
      - '8040:8040'
    depends_on:
      - ipfs
      - postgres
    extra_hosts:
      - host.docker.internal:host-gateway
    environment:
      GRAPH_NODE_CONFIG: /etc/config.toml
      postgres_host: postgres
      postgres_user: graph-node
      postgres_pass: let-me-in
      postgres_db: graph-node
      ipfs: 'ipfs:5001'
      ethereum: 'mainnet:http://host.docker.internal:8545'
      GRAPH_LOG: debug
      GRAPH_ALLOW_NON_DETERMINISTIC_FULLTEXT_SEARCH: 'true'
      GRAPH_ALLOW_NON_DETERMINISTIC_IPFS: 'true'
    volumes:
      - /var/gitopia/config.toml:/etc/config.toml

Step 6: Set Up firehose-cosmos and Graph Node

firehose-cosmos is the Firehose integration for Cosmos chains.

Configure and Build firehose-cosmos

#!/bin/bash
cd $HOME
curl -L https://github.com/graphprotocol/firehose-cosmos/releases/download/v0.6.0/firecosmos_linux_amd64 > firecosmos
sudo mv firecosmos /usr/local/bin/firecosmos
sudo chmod +x /usr/local/bin/firecosmos

Increase Default Limits

#!/bin/bash
export HOME=/home/ubuntu
export PATH=$PATH:/usr/local/go/bin
sudo bash -c 'echo "* soft nofile 3000000" >> /etc/security/limits.conf'
sudo bash -c 'echo "* hard nofile 3000000" >> /etc/security/limits.conf'
sudo bash -c 'echo "* soft nproc unlimited" >> /etc/security/limits.conf'
sudo bash -c 'echo "* hard nproc unlimited" >> /etc/security/limits.conf'
sudo bash -c 'echo "fs.file-max = 3000000" >> /etc/sysctl.conf'
sudo sysctl -p
sudo bash -c 'echo "session required pam_limits.so" >> /etc/pam.d/common-session'
ulimit -n

Configure Gitopia Node Config

gitopiad init --chain-id=gitopia gitopia-graphql --home=/var/gitopia/.gitopia
curl https://raw.githubusercontent.com/gitopia/mainnet/master/genesis.tar.gz -o genesis.tar.gz
tar -xvf genesis.tar.gz
mv genesis.json /var/gitopia/.gitopia/config/genesis.json

sed -i "s#laddr = \"tcp://127.0.0.1:26657\"#laddr = \"tcp://0.0.0.0:26657\"#g" /var/gitopia/.gitopia/config/config.toml
sed -i 's/cors_allowed_origins = \[\]/cors_allowed_origins = \["*"\]/g'  /var/gitopia/.gitopia/config/config.toml
sed -i 's/enabled-unsafe-cors = false/enabled-unsafe-cors = true/g'  /var/gitopia/.gitopia/config/app.toml
sed -i 's#seeds = ""#seeds = "[email protected]:11356,[email protected]:57656,[email protected]:11356"#' /var/gitopia/.gitopia/config/config.toml


sed -i 's#persistent_peers = ""#persistent_peers = "[email protected]:10056,c42c22d883d6ccb4cd8008dc8d5139298e631e58@gitopia.arc.peer.stakevillage.net:14256"#' /var/gitopia/.gitopia/config/config.toml

sed -i 's#unconditional_peer_ids = ""#unconditional_peer_ids = "b07d85fd94c1356d59b8352b792dbab45e225dcc,c42c22d883d6ccb4cd8008dc8d5139298e631e58"#' /var/gitopia/.gitopia/config/config.toml

echo "[extractor]
enabled = true
output_file = \"stdout\"" >> /var/gitopia/.gitopia/config/config.toml

Create Firehose Configuration File

Create /var/gitopia/firehose.yaml

start:
  args:
    - reader
    - merger
    - firehose
    - relayer
  flags:
    common-first-streamable-block: 1
    firehose-grpc-listen-addr: ":9030"
    reader-mode: node
    reader-node-path: /usr/local/bin/gitopiad
    reader-node-args: start --x-crisis-skip-assert-invariants --home=/var/gitopia/.gitopia
    reader-node-logs-filter: "module=(p2p|pex|consensus|x/bank)"
    relayer-max-source-latency: 99999h

Step 7: Configure Systemd Services

Create systemd service files for Firehose and Docker Compose.

Firehose Service

Create a file /etc/systemd/system/firehose.service

[Unit]
Description=Firehose Service
After=network-online.target

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/var/gitopia
ExecStart=/usr/local/bin/firecosmos start -c /var/gitopia/firehose.yaml
RestartSec=3
Restart=always
LimitNOFILE=infinity
LimitNPROC=infinity

[Install]
WantedBy=multi-user.target

Docker Compose Service

Create a file /etc/systemd/system/docker-comp.service

[Unit]
Description=Docker Compose graph node
After=firehose.service

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/var/gitopia
ExecStart=/usr/bin/docker-compose up -d

[Install]
WantedBy=multi-user.target

Step 8: Enable and Start Services

sudo systemctl daemon-reload
sudo systemctl enable firehose.service
sudo systemctl enable docker-comp.service
sudo systemctl start firehose.service
sudo systemctl start docker-comp.service

If everything worked correctly docker-compose logs should show something like this:

INFO Blockstream connected, consuming blocks, network_name: gitopia, provider: gitopia, component: CosmosFirehoseBlockIngestor

If you don’t see this logs and instead see errors, check the last section of this article for troubleshooting.

Step 9: Deploy gitopia-subgraph

Clone and load the ipfs files to bootstrap with genesis data.

git clone gitopia://gitopia/gitopia-subgraph
cd gitopia-subgraph
yarn
cp tests/data/*.json /var/gitopia/data/ipfs/.
# gitopia_ipfs_1 is the name of the ipfs docker container
docker exec -it gitopia_ipfs_1 sh -c 'ipfs add $(find data/ipfs -name "*.json")'

Now let’s deploy the graph:

yarn codegen # creates the required generated files
yarn build # builds wasm files
yarn create-local # creates a graph with the above give name gitopia/gitopia
yarn deploy-local # deploys the graph to the local node
# Which version label to use? (e.g. "v0.0.1"): v0.0.1

The successful output of deploying the graph would be

Build completed: QmXHtE2CPtVwzDTg1X4vMwkexZKrbiL1XDKvcSDnJd2MED

Deployed to http://localhost:8000/subgraphs/name/gitopia/feed-alpha/graphql

Subgraph endpoints:
Queries (HTTP):     http://localhost:8000/subgraphs/name/gitopia/feed-alpha

Note

Now the docker-compose logs should show that the subgraph was deployed and start syncing data for the graph.

Open the above URL http://localhost:8000/subgraphs/name/gitopia/feed-alpha in your browser and enter the query to fetch all the organizations with their attributes.

query MyQuery {

  users {
    id
    username
    userId
  }
}

You can also filter/search query using contains, eg.

query MyQuery {

  users(where: {username_contains: "gitopia"},) {
    id
    username
    userId
  }
}

This query will filter and return all the users whose name contains the string gitopia.

You can find the complementing source code here which also contains the configuration files.

Troubleshooting

Here are a few errors that you might encounter and how to resolve them. Please report to us if you find any other errors, we will keep adding to this list.

error trying to connect: tcp connect error: Connection refused (os error 111)

This happens usually because of a incorrect setup or corruption of node db. You should check the journalctl logs for more details on the error sudo journalctl firehose -f.

If the setup is correct and the issue is corruption, you can fix it by

sudo systemctl stop firehose && gitopiad tendermint unsafe-reset-all --home /var/gitopia/.gitopia && sudo systemctl start firehose

docker-compse logs -f does not show graph node indexing from firehose

Sometimes when firehose(gitopiad) takes substancial time for intial peering and starting the sync, the graph-node times out. An easy way to fix this would be to clear the docker-compose service and restart

cd /var/gitopia/
docker-compose down
docker-compose up -d

utf-8 errors in firehose logs, node halted

This is fixed by restarting the firehose service until it moves ahead of these errors.

sudo systemctl restart firehose

Conclusion

By following this guide, you have successfully set up a continuous indexing process for Gitopia using the Graph Protocol and firehose-cosmos. This robust setup ensures that your indexer remains up-to-date, even as the underlying gitopiad binary evolves with new upgrades.

With this configuration, you can efficiently index and query blockchain data, enabling developers to build and query open APIs (subgraphs) seamlessly. This setup not only benefits Gitopia but also serves as a valuable reference for other Cosmos chains aiming to implement similar solutions.

Remember to monitor the indexing process and update your binaries and configurations as needed. Continuous maintenance and monitoring will help ensure the stability and reliability of your indexing service.

- - -

About Gitopia

Gitopia is the next-generation Decentralized Code Collaboration Platform fuelled by a decentralized network and interactive token economy. It is designed to optimize the open source software development process through collaboration, transparency, and incentivization.

Website : https://gitopia.com/
Telegram: https://t.me/Gitopia
Discord : https://discord.com/invite/mVpQVW3vKE
Twitter : https://twitter.com/gitopiaDAO

A Comprehensive Guide to Running a Graph Protocol Indexer and Subgraph for Cosmos Chains

Prerequisites

Setting Up the Development Environment

Step 1: Install Required Tools

Install Go

Install Docker

Install node and yarn

Setup Filesystem

Step 2: Prepare `gitopiad` binaries and configuration file

Setup git remote helper

Setup `gitopiad` binaries for firehose

Create configuration file

Step 3: Automate Firehose Restart

Step 4: Schedule the Script

Step 5: Setting up the firehose indexer

Deploying the Graph node

Step 6: Set Up firehose-cosmos and Graph Node

Configure and Build firehose-cosmos

Increase Default Limits

Configure Gitopia Node Config

Create Firehose Configuration File

Step 7: Configure Systemd Services

Firehose Service

Docker Compose Service

Step 8: Enable and Start Services

Step 9: Deploy gitopia-subgraph

Troubleshooting

error trying to connect: tcp connect error: Connection refused (os error 111)

docker-compse logs -f does not show graph node indexing from firehose

utf-8 errors in firehose logs, node halted

Conclusion

About Gitopia

Follow us

Tags

More articles from Gitopia Blog

The Lore Token Model: Updated

Community Update: Q2 2023

Prerequisites

Setting Up the Development Environment

Step 1: Install Required Tools

Install Go

Install Docker

Install node and yarn

Setup Filesystem

Step 2: Prepare gitopiad binaries and configuration file

Setup git remote helper

Setup gitopiad binaries for firehose

Create configuration file

Step 3: Automate Firehose Restart

Step 4: Schedule the Script

Step 5: Setting up the firehose indexer

Deploying the Graph node

Step 6: Set Up firehose-cosmos and Graph Node

Configure and Build firehose-cosmos

Increase Default Limits

Configure Gitopia Node Config

Create Firehose Configuration File

Step 7: Configure Systemd Services

Firehose Service

Docker Compose Service

Step 8: Enable and Start Services

Step 9: Deploy gitopia-subgraph

Troubleshooting

error trying to connect: tcp connect error: Connection refused (os error 111)

docker-compse logs -f does not show graph node indexing from firehose

utf-8 errors in firehose logs, node halted

Conclusion

About Gitopia

Follow us

Tags

More articles from Gitopia Blog

The Lore Token Model: Updated

Community Update: Q2 2023

Step 2: Prepare `gitopiad` binaries and configuration file

Setup `gitopiad` binaries for firehose