[PROPOSAL] Rally Infrastructure & Engineering funding request

Tech/Network Upgrades
1

Hi Rally Community -

Since the launch of the Rally Network, the Rally team has been working to build out technology upgrades and infrastructure to support the continued growth of the network. Key to this growth is supporting security and scalability.

For the past year, the Rally infrastructure team has been focusing on rapidly building features on rally.io, the primary app connected to the Rally Network that has been key to bringing attention to the Rally Network as a whole. The team has also worked to support different iterations of Rewards proposed by the Rally community. We will continue to build features and iterate quickly such as building the new NFT site where we expect to get a lot of traffic.

As the project grows in popularity, engaging with tools will help the development team do our jobs faster and more effectively, as well as hiring experts to help with security and scalability/performance to get the Rally infrastructure ready for the next phase of growth. There are a few areas of focus with respect to infrastructure:

  • Monitoring and Alerting
  • Load and Stress Testing
  • DevOps Consulting
  • Economic Alerting & Monitoring
  • Anti botting / Fraud detection
  • Mobile Application(s)

Monitoring and Alerting

For Monitoring and Alerting, DataDog is a tool that the team has been effectively using on a trial basis. Many engineers have experience with it - it enables quick identification of issues and the ability to figure out performance problems in code and infrastructure. We want to be able to turn on more instrumentation so we can better understand and improve code and infrastructure. It allows us to quickly know when services are overloaded or when things are going down or not working as planned. Overall, it is a very useful tool that can provide insight into Rally systems without major engineering work.

Requested budget required for this tool: $144,000 (12 months)

Load and Stress Testing

The team needs to invest in building some tools to stress test our system. As products and upgrades are shipped quickly, and the number and popularity of creators increases, we have been simply depending on scaling the number of servers to ensure that we can handle the extra traffic. We want to prevent overload going forward and build/outsource a comprehensive suite of load/stress tests where we can discover bottlenecks and ensure various parts of our system can handle load all the way from end to end.

Requested budget required for such tools: $180,000 (12 months)

DevOps Consulting

Foghorn consulting (specifically FogOps) would be a good tool to use to supplement the efforts of Rally’s existing DevOps engineers. FogHorn has a team of expert engineers that they can deploy to work alongside the Rally team to help audit Rally systems for security, performance and to ensure Rally is using the latest technologies as well as best practices. They are experts in their field and could help to design and solve problems as needed. They have proven software that their team has built over the years and specifically used in DevOps that Rally can leverage. For example, they have developed and employed effective ways to organize VPCs and Security groups in AWS because they have done it so many times for many clients. Rally would be well served to leverage that type of expertise and software.

Requested budget required for this tool: $288,000 (12 months)

Economic Alerting and Monitoring

We are interested in investing in economic alerting and monitoring. We have noticed that while we try our best to write and test our code, there will be bugs or there will be ways others can attack us. For example, if we depend on fetching the $RLY price from Uniswap via thegraph or one of our providers, if that number is wrong, all other related calculations will be wrong. We want to invest in tools to help detect fluctuations of value in cases like these. We have identified Quantiphi as an AWS partner that could help us with Data Engineering and AI/ML. Quantiphi is a top 5 AWS partner in AI/ML, that could help set up the data infrastructure. Investing in this area will help protect the community and the stability of the network.

Requested budget required for this engagement: $432,000 (12 months)

Anti botting / Fraud detection

Addressing bots and fraud is a high priority in the interests of Rally and the community. We have heard complaints about bots or software which people have used on rally.io to scrape or get an unfair advantage against the community. Use of such bots are specifically prohibited in the rally.io terms of use, but that will not stop people from deploying them against Rally. Investing in the right tools can help detect this type of activity, or at least reduce such abuses and mitigate the risks that this will happen. For fraud detection, the network has encountered issues where people/bots are engaging in highly fraudulent activity, including use of stolen credit cards to buy creator coin. Rally would benefit from finding ways to detect and prevent bad behavior and not allow a few bad actors to affect the entire community.

Requested budget required for such tools: $180,000 (12 months)

Mobile Application(s)

Rally should begin exploring building mobile apps that would help the spread the use and adoption of Rally within the community. We see the potential of an official Rally mobile application that could help with navigating the sidechain easier than it is now with the mobile web application. A Rally mobile app could help with MFA, providing more security within the Rally ecosystem. There are several use cases for creator coins that could be a separate app, or an app that allows creators to white label their own apps - to assist them in managing their own coins/ NFTs within their community.

Requested budget required for such tools: $480,000 (12 months)

Total requested estimated budget for 12 months: $1,704,000

This budget will be requested for approval by the Rally Community from the Rally Treasury. While this total budget may seem significant, the community’s investment into building strong security, scalability, and infrastructure for Rally as outlined above will pay dividends, as creator and fan experiences will improve, and along with that, the Rally Network as a whole will continue to grow and thrive. We welcome your feedback and discussion on the outline as presented above.

4 Likes
2

Monitoring and Alerting
Seems expensive given the prices quoted on DataDog’s site. Is there any way you can tell us more about how you reached that number without telling us too much information about the backend?

Load and Stress Testing
I think this is probably worth it. We’ve been having a lot of issues with load lately, and it’s only going to get worse.

DevOps Consulting
I think this is probably worth it. It’s more than I would prefer to spend, but I would rather over spend on security, especially given that this manages finances.

Economic Alerting and Monitoring
This seems like bleeding edge tech by the price and description of the company. Is there any way we can pull it back to cutting edge? Or is there some other reason that this suits Rally’s particular needs in a way that can’t be provided by, say, a top 20 AWS partner? It just seems like the example provided would be relatively easy to solve. So perhaps there are other examples?

Anti-botting/Fraud detection
This is fine. In fact, this is an area (specifically fraud detection) I would be willing to spend more. Few things would be worse for crypto than fraud slipping through the cracks.

Mobile Application
That seems awfully expensive for a mobile app that should primarily be consuming APIs. Is that a quoted price for contract work? Or is that hiring a team of developers and building it from scratch? Because if it’s the second, I think the community would be better served by a strong API and a weaker app than vice versa; a more mobile friendly front-end could solve most issues. An alternate route that I might suggest is moving the rally.io site to React and using something like Expo to deploy React apps to native code for mobile devices; it does have a few gotchas, but it’s convenient for being able to deploy to both iOS and Android devices so you don’t have to manage two apps. And that way you would only need one cross-trained team to manage all of the apps and one team to manage your back-end APIs.

Edit: Actually I think the front page is React. Should be able to get the same team working on an app.

2 Likes
3

Monitoring and Alerting
A lot but reasonable from my point of view.

Load and Stress Testing
Not exactly sure what you want to build. Why not just use existing ones? Could you elaborate why it cost so much money to simple load and stress test?

DevOps Consulting
Just do it.

Economic Alerting and Monitoring
Are you hiring someone to do this or is this just for the tools and the consulting?

Anti-botting/Fraud detection
Just use Datadog alerts.

Mobile Application
From my understanding bonfire is building an app already or i am wrong?

1 Like
4

I see fairly broad consensus that further investments in these areas is a good idea, and only some disagreements around the proposed budgets. Thank you for laying out the case for each of these and sharing the research on cost and thinking from your team. Thanks, Mike!

I like the idea of an official Rally mobile app, but I’m concerned about how Rally would manage Apple and Google’s cut of Creator Coin and NFT purchases via the app. It may be prudent to answer questions around that before we spend too much on development of an app that could be shut down for violating their ToS. An official mobile app that couldn’t support purchases would be a step down from the current mobile web app in my mind!

Love to hear further thoughts on this, and welcome the full infrastructure and eng funding request separate the mobile app development to help the team move quickly on those important initiatives.

Cheers,

Grand

4 Likes
5

Some quick answers to some of the questions posed above before the long weekend.

  • Monitoring & Alerting: Yes, the cost estimate here is just that, an estimate, where we are trying to take into account continued growth of the network. As it currently stands, at our current usage, with Datadog only, the cost would be just 1/5th of the amount being asked for. The higher number comes from both a rough estimate of what growth may look like as well as leaving some buffer room for any additional tools/vendors in this area. This is true for most of the other numbers as well.

  • Economic Alerting & Monitoring: We are basically starting from 0 in terms of this stuff, so its going to necessarily start from just cutting edge. You are right, the example cited is not a complicated one, but this is bucket of budget is meant to cover a pretty wide range of things/tools. Including data warehouse (think Snowflake), BI tools and dashboards (Looker/Tableau etc), AI/ML stuff for detecting out of the ordinary activity, and the consulting resources to implement, connect, and build out analyses. Additionally this area will be serving a number of different teams in the organization, Product, BD/Creator Success, and Payments.

  • Mobile app: this budget is specifically to bring on a team of developers to build out a mobile app. This is not just for the “official” rally mobile app but also potentially to support a number of whitelabeled apps. The current plan for the initial version of the mobile app is to NOT support purchases and instead focus on transfers and converts. The mobile web app would continue to be supported.

4 Likes
6

Late to the conversation, but it’s very exciting to see the proposed addition of these upgrades and capabilities. These are all high caliber resources which a high caliber team has every right to demand in order to be successful and compete with maximum effectiveness. The last major startup I was in employed all of these resources very effectively. I can’t imagine us moving forward in a major way without having these in resources to deploy.

4 Likes