eosnetworkfoundation / engineering Goto Github PK
View Code? Open in Web Editor NEWA workspace for documentation by Engineering primarily regarding process
License: MIT License
A workspace for documentation by Engineering primarily regarding process
License: MIT License
Prepare for and participate in the DNS root migration of some of our key domains.
Following engineering issue 49, this ticket is to create an Automation service account on Telegram. The Automation service account will then be used to create group chats or channels, and to integrate bots in our environment. Using a service account divorces the ENF infrastructure from an employee account, reducing attacks targeted at employees and making it easier to transition infrastructure if or when employees move on. This requires a private phone number that can receive SMS messages, which is currently...challenging.
engineering issue 68 - EVM Monitoring and Alerting - Phase 1
Need AWS access for doing architecture related investigation, e.g., setup and deploy new services, work with them, take them down.
We have [email protected] It isn't for engineering support. All engineering emails from the website are going to this email. Can we create some structure that helps non-engineering ENF folks route requests?
Kevin brought up that he was seeing some anomalous results from the performance harness and suggested that if we had historical data we would be able to tell if this was spurious or not.
As part of an epic for a unified metrics and alerting system around EVM infrastructure, this ticket is to send alerts relating to the EVM AWS infrastructure health checks to arbitrary email addresses.
engineering issue 68 - EVM Monitoring and Alerting - Phase 1
Need ability to observe the real-time AWS configurations
As part of an epic for a unified metrics and alerting system around EVM infrastructure, this ticket is to write, copy, or otherwise implement a bot that alerts to an IM group chat on specific metrics. This is a generic problem and the bot will need to be able to handle arbitrary metrics eventually, but the first use-case will be to alert to IM on the status of EVM infrastructure health checks.
engineering issue 68 - EVM Monitoring and Alerting - Phase 1
Testnet: faucet.eosnetwork.com/ api-testnet.eosnetwork.com/ explorer-testnet.eosnetwork.com/bridge-testnet.eosnetwork.com
mainnet: api-evm.eosnetwork.com/ explorer-evm.eosnetwork.com/bridge-evm.eosnetwork.com
Following engineering issue 57, this ticket is to use the Automation Telegram service account to provision and integrate notification channels for the EVM testnet infrastructure alerting.
engineering issue 68 - EVM Monitoring and Alerting - Phase 1
There was a discussion on IM today where an engineer shared a utility they wrote in BASH and asked what the best way is to make their script available to the ENF community.
I believe we should adopt a pattern of publishing BASH scripts using a package manager such as bpkg for several reasons.
git clone
operations, speeding up CICD and simplifying repository structure.bpkg
handles dependencies of dependencies.For those who have worked on EOSIO before, a good package manager eliminates the complication caused by the extra eos-buildkite-pipelines
dimension of eos
, or similar BASH script repositories.
This ticket is to share an example using bpkg to publish and consume a BASH script for other ENF community members to follow. The bpkg tooling should also be added to the Tool Install Guide.
There has been recent discussion about whether a risk review be performed around the associated system provided libraries with particular focus on math oriented libraries.
We need to begin collecting EOS mainnet chain data in order to build a full-chain parallel replay system (aka "chickens" test). It may take a very long time to collect this chain data - six months or more. The first step in a chickens test is to begin collecting this data. This will also be useful to provide ENF engineers fast and reliable P2P and HTTPS API endpoints.
This ticket is to manually stand up a "sync node" to sync to EOS mainnet from public peers and collect chain data for these and other purposes.
Part of this ticket is also to create initial cost estimates for such nodes. A back-of-the-napkin estimate was $365/mo, but a more accurate and specific estimate is a deliverable for this ticket.
Would like to get AWS access for the Frontend Engineering team that is investigating GameFi. We will be doing exploratory work into new projects using the EOS-EVM and would like to deploy these projects to test environments.
Team members
What we need
Customers have not been getting email alerts surrounding EVM cloud infrastructure health check status changes. The email alerts are working with test events, so there must be something wrong with CloudWatch or between CloudWatch and the SNS topic. Debug and fix this so alerts start flowing again.
engineering issue 68 - EVM Monitoring and Alerting - Phase 1
[transferred from legacy agenda on behalf of @linh2931 ]
(Should we) add Clang build (compile only) to CICD?
Example of issue with status quo: Lambda capture of structured binding compiles in GCC but not Clang in ‘C++17’ mode. Example: AntelopeIO/leap#703
See also: Leap issue 730.
Leap or
nodeos
HTTP APIs have always been a second-class citizen. We put time and money into them, indicating we believe they meet a customer need and are important. However, we do not create reliability and stability by treating the HTTP API as a product with stakeholders, a release schedule, and versioning.For example, we recently commissioned work to change the type of compression used for one API call, and another piece of work to change the HTTP status code in the response from another API call. These were changed in-place, potentially breaking the applications of downstream consumers.
Compare this to the strategy of a normal public-facing product API such as the AWS API, Ethereum JSON RPC APIs, or really everyone else. These service providers plan their API interfaces ahead of time, develop them according to this strategy, freeze them, and release them. Changes such as the HTTP response code or data types used to deliver responses or fields of responses are not allowed. They must be planned ahead of time and changes such as this require a new API version that is released.
Eventually, older API versions are deprecated, customers are encouraged to upgrade their clients, and the old version is removed according to a schedule. Some API providers do allow new API "routes" to be added to an existing version because this does not break backwards-compatibility with clients. Most allow bug-fixes that are backwards-compatible. However, changes to existing routes are essentially non-existent.
We should follow a strategy like this to improve reliability, stability, documentation, and enable downstream consumers of our API to plan ahead.
Should we ratify the new discussion issue process?
The old meeting was not creating good outcomes and I proposed a new process. I hope to document that process in a living way as it evolves and this is the first iteration of that.
The PR is here: #29
In looking at running Doxygen as part of Leap CI, and larger discussion came up. Spoke to DevRel and they are focused on developers using EOS not developers of EOS.
The product owner determines the requirements, and helps inform the priority of the work.
Follow-on work from #12
[Transferred from legacy agenda on behalf of @greg7mdp and @ScottBailey]
(Should we) consider switching to -std=c++20 for leap and appbase? (@greg7mdp)
It’s 2023, can we have FULL C++20 support? If now isn’t the right time, when is? (@ScottBailey)
When I joined the ENF, I took notes on many of the steps I took to setup my personal Linux machine for work in Joplin. @larryk85 scheduled a meeting to discuss on-boarding, so I figured I should publish what I have somewhere before that meeting. This ticket tracks that work.
I published the following for this ticket.
I have been asked to add a TXT record to the eosnetwork.com
DNS to verify our ownership so Marketing can use Google Search Console.
google-site-verification=SBDxnXYx5--T6VszuLGvuqJJywEcU5rf2iF5WczEIiM
Marketing and Communications would like the following DNS record entered for eosnetwork.com
to authorize one of their tools to send emails from [email protected]
.
NAME scph0823._domainkey
TYPE TXT
VALUE v=DKIM1; k=rsa; h=sha256; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC4tXAIvUiae0mMLL2FE4V5pxehsZ83NIAp1KrIAFGaUZSXxTYod/xk8NVHvM++yY1vkXA7DC5hwawSV1feb1QzWmFUcKu7x8QjT5tIVoxbZJKHQb+rRO7zkGKKd1D9Cp444uG9OS9tFMuUrjpTTt7bRi6i0OA1D9h8042NXq+CvwIDAQAB
Do the thing.
placeholder for @brandenespinoza or @kj4ezj to fill in more information for
It would be great if DUNES had CI/CD support for Windows and OSX (both x86 and arm).
Currently, github doesn't seem to support nested VMs.
I believe this means we need self-hosted runners.
Recent performance run show evidence of performance issues. The TPS is varying between both versions of leap, and the type of load (transfer action, vs read only). To understand the scope of the issue and help debug we need a single big iron host. An isolated host is needed to get repeatable results.
OCI employees needs to be able to log into a host, compile and setup leap, and run performance tests.
Large host 64Gb and 16core should do 600 Gb SSD.
As part of an epic for a unified metrics and alerting system around EVM infrastructure, this ticket is to collaborate with Operations to survey their existing ability to dashboard and alert on metrics in order to prevent fragmentation within the organization as metrics and alerting are built out for Engineering. Their solutions may or may not meet Engineering needs, but we should prevent fragmentation where possible.
engineering issue 68 - EVM Monitoring and Alerting - Phase 1
Currently, all EOS Network Foundation (ENF) infrastructure exists at eosnetwork.com
or one of its sub-domains. This is ideal for (external) customer-facing services such as learn.eosnetwork.com.
Using the primary, customer-facing web real estate at eosnetwork.com
for internal services, developer work, and engineering testing creates unnecessary risk to our outward-facing properties and currently involves friction. While we have plans to eliminate the friction, the risk will remain.
There are a number of other domains marketing has picked up to protect our brand and they are more than happy to let engineers use some of them, but they have asked that their department approve changes made to these domains in advance. This is a perfectly reasonable request. Automation is grateful to work with Marketing and we are certain requests will be approved quickly. However, this is still friction.
Automation is requesting permission to purchase one or two domain names to be used exclusively for inward-facing services, developer work, and engineers. These domains would be frictionless in the sense that Automation and Engineering would have ownership over them, be able to provision resources for internal projects without risk to external customer-facing resources, and provision resources without approval from other departments. Automation has a shortlist of domain names we have confirmed are available to use for this purpose but we have intentionally excluded them from this ticket to protect their availability. The list is available via IM or email upon request.
The cost of these domains would be $20-$40 per year per domain name.
As discussed in issue 5, both Amazon Web Services (AWS) best practices and Google Cloud Platform (GCP) best practices recommend that any organization distribute their cloud infrastructure among multiple accounts, to use AWS terminology. The EOS Network Foundation (ENF) is now doing this.
ENF cloud infrastructure administrators currently juggle multiple logins to access the different AWS accounts in use. While administrators of the ENF organization do retain absolute control over all accounts and resources, there is not a clear pattern for anyone to use their existing credentials to access resources in new accounts. This creates friction for internal customers as well as unnecessary complexity and opacity for administrators. Relying on humans to secure multiple sets of credentials also increases risk exposure.
The EOS Network Foundation needs to implement federated logins within our Amazon Web Services organization. We can accomplish this with our existing resources by federating our Google Workspace (Gsuite) accounts into the AWS IAM identity center as described here and here. This will empower cloud administrators to use the AWS IAM Identity Center to grant individuals and teams access to cloud resources across the organization based on need using identity and access management (IAM). This also simplifies off-boarding by implicitly revoking access to AWS resources when access to Google resources (Drive, Gmail, GCP, etc.) are revoked.
This type of user access will be portable across the organization no matter how many AWS accounts are created for different systems. One or more select organization administrators would still be required to maintain a login detached from Google to the management account, but administrators logging in using Federation would have visibility into the whole organization by default as defined by our IAM policy.
We are going to start making decisions in a more organized fashion. We need to first decide and vote on how consensus is achieved.
As Product, I don't want to manually message Marketing and Comms teams to trigger a series of Social Media activities when a new release takes place. Because Github and Telegram will continue to be the tools we use for the foreseeable future, we should automate this process and consider whether we wish to connect this automation to additional tooling for social pushes.
Acceptance Criteria
While working on Developer Relations issue 80, I found there is not currently an organized layout of organizational units (OUs) and accounts in Amazon Web Services (AWS). These terms correspond to folders and projects in Google Cloud Platform (GCP), respectively, and we likely don't have a strategy for organizing resources there either. I brought up this concern and was asked by @ericpassmore to come up with such a strategy. This ticket tracks that subset of work.
In engineering issue 55, Automation collaborated with Developer Relations to migrate the docs.eosnetwork.com
infrastructure to a new AWS account (docs-prod
), culminating in a "DNS switcharoo." Once we are certain the new DNS records have propagated worldwide, Automation will tear down the old documentation infrastructure.
DevRel team is looking for ways to gain better visibility about tickets that are either labeled as devrel
or documentation
in both the antelope & eosnetworkfoundation orgs and also for tickets that are created on our backlog (devrel
repo)
[Transferred from legacy agenda on behalf of @stephenpdeos]
Problem: PR review assignment is not well reflected within project and standups
While presenting on bpkg as a proposal for a standard BASH package manager within the ENF at the 2023-01-10 engineering meeting, the engineering team agreed that the complexity of BASH scripts within our organization should be curtailed in favor of using "real" languages such as Python or JavaScript. This ticket is to write a document outlining indicators that a BASH script has reached that level of complexity, work should be halted, and the system should be implemented in a "real" language.
I have been asked to put this document in the Engineering wiki.
Following engineering issue 57, this ticket is to use the Automation Telegram service account to provision and integrate notification channels for the EVM mainnet infrastructure alerting.
engineering issue 68 - EVM Monitoring and Alerting - Phase 1
Need ability to observe the real-time AWS configurations
Developer Relations and Automation are collaborating to migrate the infrastructure for docs.eosnetwork.com
and stanging-docs.eosnetwork.com
to a new Amazon Web Services (AWS) account, docs-prod
. This ticket tracks the DNS migration portion, essentially Automation's part aside from providing support. Note this is colloquially referred to as a "DNS switcharoo" in the ENF.
We want to change servers for the learn portal on learn.eosnetwork.com
.
In order to do so, we need to point the DNS to the hosted zone.
I've forwarded the information to @kj4ezj in a private message, this ticket is merely for tracking purposes.
We want to launch the web IDE on ide.eosnetwork.com
. In order to do so, we need to point the DNS to the hosted zone.
I've forwarded the information to @kj4ezj in a private message, this ticket is merely for tracking purposes.
Our engineering team standard of requiring two approvals for any PR is good to have, but we need more clarity about who all is eligible to perform those approvals. We may wish to consider conditional roles for some contributors if they are the owner of a given feature set. One recent example would be to consider allowing Peter to be one of two reviewers to PRs related to performance harness since he has been technical lead on that project. A simple matrix of who is an eligible approver for what would make the review process go smoother in the future.
Ticket for tracking
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.