nebolsin Being able to search keys is good but being able to search key values is even better. In my case I use custom data for one of two things.
1) Storing locations for external data. So if I'm recording an account as a holder or issuer of a particular NFT I may record the metadata for that NFT in an external database with a marker in the issuer account's custom data. In this case the key would be unique and the data would be unknown and thus unnecessary to search for.
2) In the other case however I'm starting in an external database and trying to find a particular account by some marker value. So maybe I have marked accounts with ipfs address locations and would like to be able to find the Stellar account which matches one or more of a particular ipfs address which I have stored in custom data. In this case I would search for the value and not the key. Granted I have to search for the serialized value but that's not difficult as Stellar has the method for how they do the serialization so you just have to do that to the term before searching for it.

Hopefully that makes sense. Being able to do both has been very valuable in my own application development and while others have mentioned primarily searching key values and not the data itself both are valuable.

My primary concern at the moment is being able to filter accounts based on signers and home domains.

25 days later

Hi there πŸ‘‹ Astrograph team here. Recently we introduced a couple of useful queries to Astrograph schema:

  • filtering accounts by data entry keys and values (as @tyvdh requested):

      accounts(
        data: {
          name: "foo",
          value: "bar"
        },
        first: 5
      ) {   
        nodes {
          id
        }
      }
    }
  • querying assets from issuer’s account node:

        account(id: "GBSTRUSD7IRX73RQZBL3RQUH6KS3O4NYFY3QCALDLZD77XMZOPWAVTUK") {
          assets(first: 5) {
            nodes {
              code
              holdersCount
              unauthorizedHoldersCount
              totalSupply
              authRequired
            }
          }
        }
      }

Playground still available at https://demo.astrograph.io/graph

7 days later

Hey everyone!

We have finished alfa version of Astrologer: tool that allows to ingest or export historical data from Stellar Core to ElasticSearch.

It stores ledgers, transactions, operations with results and metas in ES and makes you able to search all the data in very flexible way using all the black magic of ElasticSearch engine.

For instance, you could easily build a query which answers the question "What was the most popular operation type three days ago in the whole network?" or "What was the daily average balance of specific account?" or "What was the biggest payment made last week?" or simply find all payments with specific memo value.

Check the repo:

https://github.com/astroband/astrologer, it contains some sample queries in Postman collection.

Astrologer can be used in isolation, as a separate tool, but we are now integrating it into Astrograph. Our first goal is to query operations from ES instead of Horizon (it's almost done), add more search criterias and make some useful aggregation results available via Astrograph GraphQL API.

Ultimate goal is to completely get rid of our dependency on Horizon, and Astrologer integration would be the first step.

We'll be grateful if anyone has any thoughts on queries or use cases which we could implement onset.

Huge news!! Seriously this is amazing. How large is an instance of a fully synced Core?

    tyvdh We're still in active development and don't have final numbers for fully synced core. Currently in progress of setting up production Kubernetes cluster on GKE, so should have better estimates for resource and storage requirements in a few weeks.

    For development we fit into single DigitalOcean's s-8vcpu-32gb droplet (the specs are 8 vCPU / 32GB RAM / 640GB SSD), which runs everything:

    • stellar-core in validating mode, which is backfilled without gaps from ledger 20kk to the present moment, so it has all the info for roughly 4 million recent ledgers. It uses local DB and publishes to local history archive (archive is currently 100GB)
    • postgresql (core DB size is 207GB, biggest tables are txhistory β€” 155GB, txfeehistory β€” 36GB, scphistory β€” 3.5GB).
    • elasticsearch (currently 155GB total, but it might be bloated with some old dev experiments, etc)
    • astrologer ingester, which constantly feeds new ledger data from core to ES
    • astrograph-server which serves our demo GraphQL
    • astrograph-ingester which feeds realtime subscriptions with new data from core's postgres
    • prometheus collects metrics
    • hasura and postgraphile instances targeting the stellar-core DB
    • some other stuff
    7 days later

    @dzham @nebolsin +1 for abilities to query signers, home domains, inflation stats.

    I think support for queries on some of the core data that is not available in horizon, might give developers more motivation to use this kind of tool.

      4 days later

      gleb Hi! Can you please elaborate, what do you mean by "some of the core data"?

      6 days later

      @nebolsin

      Interesting project. As my apps are client-side, I'm an heavy Horizon API consumer. I'm sometimes frustrated by Horizon limitation so I hope something better could be built.

      For me to switch, it would require that at least all Horizon data can be fetch from your API. For instance, I could not retrieve the account signers type, which is an important information. (Of course I understand that it is work in progress).

      If I can give you an advice, it would be to make sure that Horizon can be emulated on top of your API. From there I would expect devs to start adopting your solution, and even nodes to replace Horizon.

        MisterTicot Thank you for the feedback. We're aiming to make Astrograph at least as capable as Horizon, meaning that if a task can be solved with Horizon, it should be also solvable with Astrograph. The lack of signer type is a good catch, and we'll fix it promptly. In the meantime you can easily find out the type from the signer string itself: ed25519 public keys start with 'G', pre-authorized transactions start with 'T' and hashes of the preimages start with 'X'.

        Regarding Horizon emulation we're currently considering an option to introduce a small GraphQL service which will use Horizon's REST API as a datasource. That's how Astrograph currently serves historical queries while we're finishing its ElasticSearch ingestion engine. Then we can either include this API into Astrograph main schema via GraphQL schema stitching, or leave it as a separate endpoint.

        What the hell is an open-source GraphQL server for Stellar network?

        You have to explain it in twenty words or less because I don't like to read large blocks of text, it scares me.

        nebolsin changed the title to Astrograph β€” Explore the new horizons of Stellar network .

        Hello, folks! Astrograph team here πŸš€ Some quick project updates:

        1) We added our own payment paths finding mechanism. We don't use Horizon for it anymore. Here how it looks:

        {
          findPaymentPaths(
            sourceAccountID: "GA4DMQ3VSHIVROQ42PJVJOD7X4PYT5BXAWV672CAWOWIADXC3RGZEOMZ",
            destinationAsset: "USD-GBSTRUSD7IRX73RQZBL3RQUH6KS3O4NYFY3QCALDLZD77XMZOPWAVTUK",
            destinationAmount: "0.3"
          ) {
            sourceAsset {
              id
            }
            destinationAsset {
              id
            }
            sourceAmount
            path {
              id
            }
          }
        }

        There are some differences with Horizon output. Astrograph returns only one best path for each asset in source account's trustlines (plus native lumens). Also, it doesn't search for "detour" if the source account has destination asset in its trustlines.

        2) We drop Horizon usage in serving orderbooks too. The schema stays the same


        Astrograph still has a playground at https://demo.astrograph.io/graph. And we still would love to hear your feedback πŸ™‚