Stellar NebulaForge - "One-click" deploy stellar infrastructure to the cloud

talawahtech

My entry to the build challenge is designed to significantly reduce the barriers to entry for anyone looking to get started with Stellar by allowing them to quickly and safely deploy their very own nodes in the cloud.

The plan is to target a wide array of use cases, from newbies looking to quickly get started by playing around with an ephemeral node, all the way up to anchors looking to deploy production infrastructure.

My initial focus will be on AWS and I will start with a minimal single-server deployment that runs the stellar/quickstart docker image.

The infrastructure will be provisioned using AWS CloudFormation templates which allow you to initiate a deployment on your own AWS account by simply clicking on a link.

My ideas for the roadmap include the following. I am open to feedback and suggestions from the community. As more functionality is added, separate templates will be created, allowing users to select the template that best fits their needs.

Minimal epheremal quickstart - test network, single VPC, single AZ
Use RDS/Aurora for Postgres instead of built-in option
Split stellar-core and horizon into two separate containers/servers
More complex public/private VPC setup, multi-AZ failover
Multiple horizon instances w/ load balancing
Multiple participation profiles - watcher, archiver, basic validator, full validator
Additional software - Federation server, Bridge server, Compliance server
Test harness for validating the infrastructure
Logging, Monitoring, Alerting and other best practices
Option to se EKS (Kubernetes) instead of ECS when it becomes available
Deploy to Google Cloud or Azure using their native toolkits (or terraform)

All code will be Open Source and available on Github.

Gitzman

I will absolutely use this. Thanks!

talawahtech

The first NebulaForge template + documentation is now available. Please give it a try and let me know what you guys think: https://github.com/starformlabs/stellar-nebulaforge-aws/tree/master/quickstart

The "one click" links are in the README.

talawahtech

For those who don't need to RTFM:

US East (N. Virginia) - Launch Stack ?
US West (Oregon) - Launch Stack ?
EU (Ireland) - Launch Stack ?
EU (Frankfurt) - Launch Stack ?
Asia Pacific (Tokyo) - Launch Stack ?
Asia Pacific (Mumbai) - Launch Stack ?

talawahtech

This first NebulaForge template has been renamed to "quickstart" and received the following updates:

1) System logs from the ec2 instance (e.g /var/log/messages) are now captured and sent to CloudWatch to make it easier to troubleshoot issues without SSHing into the instance

2) An Elastic IP is automatically created by the template and assigned to the instance at startup to ensure a consistent IP is maintained across reboot or failures

3) The DB password is now automatically generated and stored in the SSM Parameter Store instead of being output directly to the logs

More details and launch links: https://github.com/starformlabs/stellar-nebulaforge-aws/tree/master/quickstart

talawahtech

The second NebulaForge template is called the "single-ec2". This is similar to quickstart in that it runs on a single instance, but each component (core, horizon, postgres) is now running in its own container using these cloud-optimized docker images

Running each component in its own container is more in line with best practices and allows us to send more detailed core/horizon logs to AWS CloudWatch so that the user to review them directly from a web browser.

More details and launch links: https://github.com/starformlabs/stellar-nebulaforge-aws/tree/master/single-ec2

talawahtech

The third and most interesting template so far is creatively named "single-ec2-single-rds". For this template stellar core and horizon are running on the same instance, but RDS is now used for the database and local state is stored on an EFS backed volume.

This means that data is retained even if the ec2 instance fails or is terminated. Only rudimentary forced-termination testing has been done so far, but in all cases core and horizon simply picked back up where they left off and continued processing after a new instance was automatically launched to replace the terminated one.

More details and launch links: https://github.com/starformlabs/stellar-nebulaforge-aws/tree/master/single-ec2-single-rds

This is the first step towards building some more reliable and robust templates. If anybody wants to start trying them out and giving me some feedback, now would be a great time.

Torkus

This is such an excellent job, kudos! As I have never used AWS before do you have an estimate of how much it would cost per month? Just an approximate for basic querying capabilities.

talawahtech

Torkus I've just updated the READMEs with more detailed cost information.

FYI new AWS accounts are eligible for a 12 month Free Tier benefit which means any of the templates that I have posted so far would cost you less than $1/month (depending on your bandwidth usage) if you use t.micro instances.

Beyond that the cheapest single-ec2 option is about $8/month and the cheapest single-ec2-single-rds option is about $24/month

Also remember that AWS is Pay-as-you-go, so if you are just testing you can just run the instances on demand by deleting the stack when it is not in use. It will have a limited amount of history available every time you launch it though.

I plan on looking into how feasible it will be to use db snapshots to launch with existing history.

StatCoin_Team

We might be interested in your cloud deployment. Send the specifics to team@statcoin.io so we can discuss internally. It would be nice to move our node to AWS before we reach full circulation of our coin.

Information on StatCoin at www.statcoin.io
Created by the Statistic Brain Research Institute
www.statisticbrain.com

savebattgo

Would love to use this if it goes live. But we will need manage services