edit: woops large post: tl;dr: what would you like to see do me better than stellarbeat.io does? also, send me your SCP explanation resources! I got interested in the SBC project idea of building something that gets a little more useful than stellarbeat presents. The first step was getting information from stellar-core, so I built a stellar-core client, and I now have access to I think anything that's possible to get from SCP nodes. The next step is figuring out what data to actually get, or maybe how to process and present it in a way that we're actually learning something useful. The basics that stellarbeat already covers are quantities, like how many validators are there, what quorumsets are they in, what are their names, where are they located, what are their peers. Perhaps we could get a little jiggy and try and dig out more information about those nodes, maybe we could even build something new that actively builds information on validators, but that would be a project in and of it self. Definitely a possibility though. For the basic premise though, how do we get some quality from those quantities. Maybe we could measure response times? Could some validator sets be more productive than others? If there's 80 validators in the network, but 10 validators actually decide on the 99% of the transactions, and 5 of those are IBM (?), maybe that's an indicator we're not that decentralized? Not saying that's the case, just saying it might be interesting to show something like that. I have to admit that I'm just dropping in the stellar community, and have not been up to date with historical events , have there maybe been moments where the network ran into capacity or reliability issues that we might indicate or even predict with the right metrics? Maybe some kind of netsplits that have seperated large groups of validators? I have been reading the paper, and watched a couple SCP videos, but I'm not 100% firm yet in my understanding of how quorumsets are formed/disbanded and how transactions are submitted to quorumsets (I think anchors would submit them to full nodes through the web api of one?) If anyone has a nice resource to really cement my understanding that would be very helpful as well, in the meanwhile I think I'll re-read the paper a few times and perhaps browse some more through the stellar-core source. Thanks!

Hi, as I don't run a node, but needed a core-dump of all accounts for a certain ledger I had problems to get access to it. Thanks to @Cajga I got the dump now, he helped me with that. I guess it would be good to have easy access to any ledger dump. Maybe this could be one service you could provide?

That's interesting, I think I saw your thread, you just wanted the balance for all known addresses right? I'm not sure if there's a way to get that without simply processing the entire history. Doesn't stellar have chain explorers where you can just query any address balance? Or do you really need the entire DB at once?

I think the way you got it is the easiest. There's a tool called stellar-archivist that is easy to modify (as it's in golang) to implement some functionality you might desire. I'm not sure what a general use case for this would be though. What sort of scenarios would people need ledger dumps for?

Hi, I needed it to prepare an airdrop on stellar holders. So the usecase is not so comon. Still I think it would be good to have the data as transparent and accessible for everybody as possible. Thanks for the archivist link, I will checked it.

A few ideas: A GUI for Stellar-core (a bit different than what you are going for but a similar premise) Something like this for Stellar: https://bitnodes.earn.com/nodes/leaderboard/

Are you actively working on the idea? I'd be interested in collaborating. My main motivation is to fully grok the whitepaper and creating a better visualization of the real network and quorum graphs seem like a fun way to do so and also get up to speed with some of the codebases and protocols. https://stellarbeat.io/ is off to a good start but I think we can do better on the visualization side. They created a custom crawler https://stellarbeat.io/howitworks to get the relevant info. Would be nice if they open sourced it. I wonder if the crawler is totally its own thing, or just a tweaked version of Stellar Core that stores more info / tries to connect to every node etc. They offer a dump of their data https://stellarbeat.io/nodes/dataset so might be a good headstart if you want to focus just on the visualization aspect. The way I was imagining the project was in 3 parts (not sure if I'll actually work on any of them): 1) Augment stellar core to collect additional data about nodes / network quorum and possibly more actively query all nodes if needed. Not sure what data is possible to get but start with data similar to stellarbeat. (Could skip this step and just use data from stellarbeat directly). I wonder if historic or changes in node / network quorum data will end up being interesting. 2) Augment horizon with apis to get data about nodes and quorum from an augmented stellar core. 3) Create an iOS app that visually displays the quorum graph. iOS because I like it ? Some ideas: - show selected node in center with it's incoming and outgoing 1st and 2nd degree connections concentrically outside and overlay quorums info as shaded area or lines - show some of the concepts in the whitepaper like Dispensable Sets or at least the most relevant ones near a selected node - do some pagerank like calculation on nodes to determine their importance and thus size (need to grok the SCP better before figuring out what really makes sense to show) - could get into the details of specific iteration and show nominations and votes on ballots for each node or how it propagated (maybe this is not knowable tho)

Definitely a good plan to get a deeper understanding of the white paper -- those are fantastic ideas! I am actually swamped in side projects, so these two ideas were more things I would love to see someone do, but not quite something I have time to do myself ?.

rob I really like that leaderboard, the way they calculate the peer index is inspiring, that might actually give us some semblance of a quality ranking for validators. I'm not sure about the gui for stellar-core, what sort of things would one do in that gui? Would it simply be a wallet that's connected directly to the core network, sort of like bitcoin wallets are? jeffditullio Yes! Actively working on this. You can follow my progress here: https://github.com/tinco/stellar-core-go . I still have to add a README and the BSD license to it, it sort of evolved from me playing with the API to a full stellar-core client. It's already usable as a stellar-core client library for Go projects. There's two example clients in the project as well, one that gets lists of peers from nodes, and one that gets quorumsets from nodes. There's also a Ruby script that crawls the entire network for nodes that accept incoming connections, which produces a list quite similar to stellarbeats' list of nodes. So for your step 1, I've already made good progress. I don't like C++ so I didn't opt for modifying stellar-core, instead I just reimplemented it in Go. Using the stellarbeat data is also a good option, they've got a nice little api endpoint that provides the entire dataset. Step 2) that's interesting, the way I understand it Horizon has a little bit more higher level information about the network, I'm not sure if it could tell us more about quorumsets, but there's bound to be some useful information in there. Step 3) I personally don't like iOS development at all, so if you're enthousiastic about it, definitely use stellarbeats or my data to make an awesome app, I'm sure people will appreciate it. I've spent some time on slack and keybase talking to people about what's useful to show in a GUI, and what data is useful to collect. I'll try and come up with a sort of summary on that so you can use that as well.

@tinco I'm not sure about the gui for stellar-core, what sort of things would one do in that gui? Would it simply be a wallet that's connected directly to the core network, sort of like bitcoin wallets are? I recommend checking out the slick TUI (terminal ui) used by grin . Essentially, as one running a node, I'd love to have a nicer way of seeing my quorum config, the peers I am connected to, and general analytics tools. Imagine you are running a node as a business and relied on this node for serving customers and making money -- having a nice way to see data about the health of your node is critical.

Hi Tinco, Jeff, Rob and Tim! Dieter from Stellarbeat here. Thanks for a very interesting thread and great feedback! Maybe it's helpful for me to provide some insight in what we're currently working on: - we're open-sourcing our Javascript crawler/listener in the next few weeks - we're working on visualising more information on network health: are there single points of failure? Are we all just reliant on SDF Validators? Can we detect and alert when there's a lack of quorum intersection? - we want to dig deeper in specific node reliability too, to hopefully in the future give network participants a chance to make informed decisions when choosing their peers. Including more fine-grained statistics on uptime, reliability, latency and even security. Ideally we'd come up with some kind of 'trust index' that we can sort nodes on. Feel absolutely free to use our data ( https://stellarbeat.io/nodes/dataset ) to build on top of, or improve, anything you want. That's why it's there! :-) And if there is any feedback that you want to share, please keep doing so. That's how we learn. Let me know if there's anything else we can do to help. Cheers!

What are indicators of a healthy Stellar network?

rob

@jeffditullio Woah nice, looks great! Will have to spend some time this weekend digesting the math

tinco

Hey peeps, just writing in that part of the project, the Quorum Explorer site is now online, you can visit it at: http://www.quorumexplorer.com.

It does not have all the functionality yet, right now it's just a dump of info from the crawler, plus some small transformations to get useful things like the concept of organizations.

Already some validators have played with the data and gained some insights, so I think the result is promising!

jeffditullio

tinco cool, digging into your site now. Nice domain name! Here's what I came up with:
AppStore
GitHub

Freitag

Looks like you have a json.parse error that is blowing up the whole site.

tinco

Just a small announcement, starting today I'm on a full time job for 2 weeks, so development will be a bit slower (even slower, than when I was having a holiday ?). I have started on the health page, I hope I will get some time for it this week to get an initial version done.

jeffditullio

tinco a full time job ? hehe I see you're making some progress on quorumexplorer.com though! Let me know when you're at a point you can share your data and I'll integrate it. I'm curious to see the differences between yours and stellarbeat data. Also I through together a rough react.js site https://nodestar.info/ ... web front end is not my thing as you can tell but it was fun learning react.

tinco

Data is now available for ingestion: http://api.quorumexplorer.com/v1/nodes.json I haven't documented it yet, but it's straightforward. I gave it version number v1, so you know it will keep being available in this format.

There is also /nodes/${node_id}.json for more information on each node, and /nodes/${node_id}/history.json for full history on each node.

Maybe at some point there should be a graphql api, but for now just some straight forward json api's will have to do ?

jeffditullio

tinco

@tinco thanks for the api. I just started digging into it. Questions/feedback:

Is there a way to tell participation level (watcher/archiver/basic/full)? https://www.stellar.org/developers/stellar-core/software/admin.html#level-of-participation-to-the-network
Looks like you get org info from known validators list. This seems useful for informational purposes in exploring but not reliable for calculating trust or resilience as it's easy to have many nodes not listed or listed under different info which would seem unassociated.
Is accepts_connections just the most recent value? Looks like some nodes where that's false have true in history? If so seems more useful to show an average and then the full history.
trust_table is interesting. Looks like = 1.0/threshold w/o inner quorumset and =1.0/threshold/innerQS-threshold if the node is in an inner quorum set. There might be a bug enhance_accounts line 29 see the last 3 table values for validator GCKWUQGSVO45ZV3QK7POYL7HMFWDKWJVMFVEGUJKCAEVUITUCTQWFSM6

Imagine a case of a validator with qs like:
root-qs (threshold 1 of 2)
inner-qs A (threshold 2/2: nodeC, nodeD), inner-qs B (threshold 1/1 nodeD)
This is equivalent to quorumset of 1/1 of nodeD

AKA
( C && D ) || ( D )
simplifies to
D
but in your forumula you're giving trust to C. Although in my analysis no validator actually reuses a node in its quorum set.

Also, should trust be the same for a node that is in qs (threshold 1 of 1 nodes) vs qs (theshold 1 of 10 nodes).

This is the reason I came up with with:
https://github.com/foundero/NodeStar#math

trusing_nodes - this is what I'm calling direct incoming. Also consider indirect incoming where you count all validators that indirectly trust this validator by following each quorumset node's quorumset. I think this get more towards something like pagerank. I also calc indirect outgoing... these show emergent properties of the network which I'm calling clusters (kinda like tiers) where each represents a set of validators with identical indirect incoming+outgoing sets. Interestingly there are not very many clusters, so it makes the entire network a bit easier to visualize: https://nodestar.info/#/clusters/1

hit me up on keybase/slack if you want to discuss

jeffditullio

I added a quick integration with your data https://nodestar.info/ see the top left toggle. You might need to do a hard refresh. I'll spend some more time on it later today hopefully. Feature request - please add quorum set hash key to your data as I had to create some fake ones to get nodestar working. Thanks Cheers!

Freitag

The api is a great start! May be interested in integrating this data into my 3D quorum explorer as it provides more structured data than stellarbeat.

« Previous Page