Below is a log of notes for the MonkeDAO Validator, and all the resources needed to start one.
Validators 1/2 As of December 20th 2021. MonkeDAO operates 2 validators. - 1 Mainnet validator that is managed on our behalf by Genesys Go. (Validator as a Service) - 1 Testnet validator that is running in an Equinox Data Center in Madrid, Spain (Bare Metal)
The reason for this is quite simple. When the decision that was made, to have run a validator. The technical knowledge of "how to set it up" was missing from the team. Genesys Go offered an easy route to get set up as the first DAO run validator in the ecosystem.
There is however a drawback to this. For new validators entering the ecosystem. There is a "delegation program" available from the Solana Foundation. As a validator operator. You are not eligible for the delegation if your server is managed by a 3rd party.
This delegation program can be quite bountiful in terms of staked SOL (which earns rewards, just like any other stake). Typical allocations can reach upwards of 40k SOL. With some of the bonus delegations for performance, this number can 2x to 80k SOL staked.
There is a nuance here to the stake getting allocated however. To be eligible for the stake delegation. You need to "prove" that you are able to run and manage a performant validator on your own. The idea behind the testnet validator was to have a playground where learning could be done and a "new" validator tested, that would not affect the rewards that the current validator is collecting. This validator, once live on testnet, needs to reach certain performant metrics that are set out by the Solana Foundation. These important metrics, are all captured on the dashboard pinned in this channel.
There are 3 more caveats. - The staking delegation CAN take up to 2 years to activate. - Your validator on testnet needs to have enough SOL staked (More than 5000 SOL), to accumulate leader slots. - Your validator needs to be performant for at least 5 of the last 10 epochs.
So, there are a ton of considerations and steps that need to be taken in the right order, for all of this to work. One of the biggest considerations, once the validator has been activated, is "how do we get 5000 testnet SOL".
1.) There are testnet tools that allow you to airdrop yourself a maximum of 1 SOL per command executed. (You use a CLI to access the server). You are rate limited to the amount of SOL you can airdrop in a given period to the validator too. I gave up after 35 SOL airdropped over 2 days. Also, the Solana foundation explicitly states that you should not accrue the required 5000 or so SOL in this way. 2.) You can find a 3rd party that is willing to send you a large stake. This is the route we ultimately went down. A helpful member of the Solana discord sent 10k SOL to our testnet address. However, this took a couple of weeks to organise. There was also an issue, in that the wallet given for transfer, was not our "main" wallet on the validator. This issue took another 3 epochs (1 week) to rectify as there was an uneducated monke.
The testnet clusters and network often goes offline. Restarts and updates are required very often. OS updates are needed weekly.
Mainnet validator (https://bit.ly/MDMainnet) Is up and running and above breakeven. Still managed by Genesys Go. That will continue to happen until we transition the testnet validator to mainnet. Once the testnet validator has been moved over. We will transition all of the staked SOL, from the GG validator, to our bare metal validator. (This is a single command, which executes and moves everything over). We currently don't have a dashboard to monitor the performance of the mainnet validator.
Testnet validator (http://solana.thevalidators.io/goto/kMkag0Tnz) Is up and running and purrrrring along. A 5000 SOL stake is currently being activated and will be staked by December 23rd 2021, earning leader slots and enough rewards to show us as extremely performant for the required 5 out of 10 epochs. Realistically we need 1 full month, of no validator downtime and active testnet staking to be eligible for the Solana Foundation delegation. December 23rd will start day 1 of our 1 month "performance run". We are working on reducing the "potential 2 year" timeline. However, this is an extremely sensitive matter and the "how" is not something that will be divulged right now. Rest assured, we do not anticipate running both validators for too long into the new year.
Once the validator has been moved off testnet, on to mainnet. We will need some additional volunteers to go through training. There are a ton of *"to-do" * items on the validator. Any dev or linux proficient DAO members should reach out and nominate themselves for a crash course in all things validators. We will need to have a team (3-4 people) across different timezones that have the knowledge and capability to restart, maintain and manage the validator, if something goes wrong
Documents, consoles, dashboards, scripts and required Installations
Required Linux dependencies Rust | OpenSSL | Libudev\
sudo apt update && sudo apt upgrade -y | Update server
solana balance | validator operation wallet balance
solana airdrop 1 | airdrop operation wallet 1 SOL
solana config set testnet | set devnet/testnet/mainnet-beta \
solana catchup ~/validator-keypair.json | catchup to Solana network
solana gossip | see list of current validators and RPC nodes
sudo systemctl stop solana-validator | Stop validator
sudo systemctl start solana-validator | Start validator
sudo systemctl restart solana-validator | Restart validator \
sudo systemctl status solana-validator | Validator status
tail -f /mnt/solana/log/solana-validator.log | Check validator logs
solana -k authorized-withdrawer-keypair.json withdraw-from-vote-account ~/vote-account-keypair.json ~/validator-keypair.json 1 | Moves rewards from voting to main wallet funding validator operation. Number at end changes withdraw quantity