Rally.io Technical Incident Policy Discussion

Community Treasury
1

Hi everyone - wanted to start a conversation about how to address technical incident.

The Rally community and network are moving extremely fast to implement and develop new capabilities that better serve creators, fans and crypto communities. As with any fast paced development, the rapid introduction of major new features, testing, security and scalability can increase risks for the network. It is a priority for the community to ensure that the innovations being achieved by the Rally network are not putting the Rally community or the network at risk.

As the community is aware, one key part of the Rally Network is Rally.io, known as the “sidechain”. On the sidechain, the core development team may make technical and infrastructure adjustments on behalf of the community. If any inadvertent incidents occur as a result of such adjustments, a clear policy on how to address such incidents would enable the community to minimize negative impacts. In addition, in case of an external attack or hack affecting Rally.io, the community should be prepared with policies on how to limit exposure and compensate, in whole or in part, any losses suffered by creators and community members affected by such incidents.

To address these types of incidents, this discussion thread is posted to start a discussion around implementing a Rally.io Technical Incident Policy that will outline the process to address any incidents that might result from technical adjustments or external impacts on the Rally.io sidechain. A draft of a policy is presented below for discussion, and once feedback is received, the final version of the policy will be posted to Snapshot for a community vote.

In summary, this policy:

  • Provides the objectives for the policy
  • Grants authority to the parties to execute on the duties and responsibilities required to implement the policy
  • Outlines the types of incidents and processes for response
  • Outlines reporting, rewards and governance with respect to the policy to ensure transparency for the Rally community.

Please join the discussion and provide your feedback and thoughts on this objective and the draft policy.

Rally.io Technical Incident Policy

Purpose

This Rally.io Technical Incident Policy (the “Policy”) establishes Rally community policies and guidelines governing technical incidents affecting the Rally.io sidechain.

This Policy:

  • Determines the delegation of duties and responsibilities by role
  • Outlines incident types, actions, reviews and approval processes
  • Defines governance
  • Outlines reporting requirements

Delegation of Duties and Responsibilities:

The Rally community is responsible for approving this Policy through a community vote. Through such approval, the community authorizes the Genesis Team to appoint a Sidechain Administrator. In addition, the community authorizes the Sidechain Administrator and Community Treasury Manager to take actions necessary to ensure oversight and review of technical incidents, assess potential impact to the Rally community and network, take actions to correct incidents found, outline the financial impact and report on the findings and outcomes consistent with this Policy.

Role Responsibilities:

Sidechain Administrator:

The Sidechain Administrator is authorized to outline found incident details, including the cause of such incident (e.g. internal issue or external origination), outline steps to correct the incident and mitigate the damage, outline financial impacts from the incident and identify recommended remediation efforts to minimize any adverse impacts to community members.

In addition, if the Sidechain Administrator becomes aware of an incident, the Sidechain Administrator is authorized to temporarily halt all sidechain activities, and suspend accounts that are identified as being involved in the incident, until the full extent of the incident is reviewed and a plan is implemented with respect to next steps. The Sidechain Administrator may restore sidechain activities and any suspended user accounts, when, in its reasonable discretion, it determines that the incident no longer presents a risk. For purposes of this Policy, the “incident period” is the time period starting at the time that the incident is deemed to have commenced by the Sidechain Administrator and ending at the time that the Sidechain Administrator deems that the incident no longer presents a risk.

Community Treasury Manager:

The Community Treasury Manager is authorized to review with the Sidechain Administrator details of each incident and must approve recommendations on how to correct incidents prior to the recommendations going into effect. The Community Treasury Manager, or its designee, is also authorized to communicate information about the incident and steps taken, or anticipated to be taken, to minimize adverse impacts to the Rally community and network.

Incident Types, Actions And Review Process:

  • Incident(s) resulting from sidechain issues (e.g., bugs, technical issues) - Once an incident is found and fixed, the Sidechain Administrator will also identify and record all wallets impacted and the extent of the impact, and upon review and approval by Community Treasury Manager, restore the sidechain and any suspended user accounts.

    • During the incident period, if a sidechain Rally.io account bridges out an equivalent value of $RLY 10,000 or under to a mainnet wallet, no further action will be taken to recover such funds
    • During the incident period - if a sidechain Rally.io account bridges out an equivalent value of over $RLY 10,000, the Sidechain Administrator may take reasonable actions to recover any balance over $RLY 10,000, including outreach to such account holder and the offer of a reasonable reward to be funded by the Community Treasury and granted to such account holder upon the return of the larger balance.

  • Incident(s) due to external origination - When an incident is found that is the result of external origination (e.g., malicious hacker or attack, security breach), the Sidechain Administrator will identify and record all wallet(s) impacted and the extent of the impact, and upon review and approval by Community Treasury Manager, restore the sidechain and recoup any and all funds impacted.

    • External attack that benefits other wallets -
      • The Sidechain Administrator is authorized to take reasonable actions to recoup funds over $RLY 10,000 from benefiting wallet. The perpetrator’s sidechain account(s) will be suspended from the sidechain and all benefiting wallet(s) that do not return the funds within the defined timeframe will also be suspended from the sidechain.
    • External attack that benefits the perpetrator -
      • The perpetrator’s Rally.io account will be suspended and, if appropriate, terminated, and reasonable actions may be taken to recoup improperly gained funds.

Reporting and Rewards:

  • Community Reporting and Rewards - In the best interests of the Rally.io sidechain and the Rally network as a whole, members of the Rally community should highlight any incidents before bridging out. If an incident is identified, community members should notify the Sidechain Administrator of such an incident via email to security-ext@rally.io. Rewards for reporting of incidents will be granted by the Sidechain Administrator, after approved by the Community Treasury Manager, in its reasonable discretion, based in part on the severity of the incident, the amount of benefit gained by the community due to reporting, and other factors.
  • Rewards Funding - In order to fund rewards, mitigate negative effects caused by lost or improperly bridged $RLY, and other matters, the Rally community authorizes the funding of a Technical Incident Fund (the “Fund”) from the Community Treasury. Initially, the total Fund balance will be set at 1,000,000 $RLY. As such funds are depleted in accordance with this Policy, the Community Treasury Manager may authorize additional $RLY to be transferred to the Fund wallet address, such that the Fund holds up to a maximum of $RLY 1,000,000 at any given time.
  • Governance - Quarterly, the Sidechain Administrator and Community Treasury Manager will provide a report summarizing all incidents and resolutions, including all spends from the Fund, within 30 days following each calendar quarter on Discourse.
1 Like
2

This seems like a great starting point. I notice we have side stepped using the words like legal or legal action is this what you mean reasonable actions?

Should we not give the community treasurer the right to freeze funds in a particular area if there is hints of wrong doing? Sometimes this needs to be done quietly and does not need the whole community involved. Than after the investigation there should be a full report distributed to the community which should contain the process, actions, outcome and conclusions.

I would like to see some comments from someone with some expertise in this area

3

Great to see the proactive approach; please include what will be the official channel(s) for sharing and updating information to the community on any incident.

1 Like
4

Hey @sixmofo, thanks for the thoughtful response as always. Within the governance of a decentralized community, I think the removal of the assumption of “legal action” as a default is appropriate. Unlike a situation where there are contracts between parties or a central controlling entity, what runs a healthy decentralized community are the community members themselves (both taking actions to protect the network, and approving policies to address potential needs proactively). This policy is intended to govern specific technical incidents, whether unintentional results of an internal update, or an external action. In either case, the appropriate response may differ - in some cases, it may be outreach to cooperative community members, and the granting of a reward for good actors, and, if recourse is appropriate (e.g. in response to a hostile attack), investigation and legal action can be pursued. Approving a technical incident fund, as suggested in the draft policy, can enable all these actions, with discretion exercised by the sidechain administrator and community treasury manager.

I think your suggestion of freezing funds in a particular area as warranted in the case of suspected or proven wrong-doing is a good option. And, as you suggest, with the authority granted by this policy, the sidechain administrator can take such actions within its authority. That option can be built into the policy for clarity, along with other suggestions of potential steps to be taken in response to wrongdoing if appropriate, or, as drafted, left in the discretion of the administrator and treasury manager.

And as you recommend, the last bullet point of the suggested policy includes a governance and reporting obligation, through which the administrator and treasury manager will summarize any incidents as well as actions taken, as well as have accountability on describing any uses of the fund. While this policy authorizes actions taken on behalf of the community without specific approval in each instance (both for expediency and privacy to handle immediate needs), such authority must be accompanied by transparency in reporting. If this policy is approved, changes to it as merited can also be adopted, once the community determines if this initial approach is working.

As always, the Rally community is designing and implementing governance that riffs off of best practices from other decentralized, and traditional, communities, but customizes it to fit the needs of the Rally network. Any other feedback and suggestions are welcome.

1 Like
5

Hey @Robb, thanks for the comment and the question. The current thinking is that reports will be made via Discourse on a quarterly basis to disclose all actions taken pursuant to the policy during the prior period. To the extent that incidents occur where it would be appropriate or necessary to alert the community, the announcements channel on Discord will be use to share and update contemporaneous information. In some cases, due to the ongoing nature of the investigation or other security or privacy priorities, the whole community may not be updated until all necessary actions have been completed. If appropriate, an update will be provided then, or else the quarterly report will include a description of the incident, actions taken and a review of the outcome.

1 Like