Building Your Own Data Center: Railway’s Journey
Since its inception, Railway’s computing infrastructure has relied on Google Cloud Platform. However, this dependency posed several challenges, risking the business’s sustainability. More critically, using a hyperscaler restricted the delivery of an optimal platform for customers, impacting pricing and service quality. A significant drawback was the limited assistance received from providers, despite substantial annual expenditures. Consequently, Railway initiated the Railway Metal project last year, starting with the first data center site in California. The project aimed to provide customers with an “infra-less” cloud experience, necessitating expertise in establishing physical infrastructure.
The Launch of Railway Metal
Commencing in January 2024, it took Railway five months to activate the first servers and an additional three months to onboard users. The primary step was securing a space. Opting for Cage Colocation, Railway leased a private area within a provider’s data center, offering a secure, customizable environment. Space costs relatively little, but power and cooling expenses can vary significantly by geography. Railway established a fixed monthly power commitment to ensure availability on demand, addressing critical redundancy needs with independent power feeds.
Network Infrastructure Considerations
Achieving minimal latency necessitates solid global connectivity. Railway selected data centers with connections to Tier 1 ISPs and Internet Exchange (IX) participation. For optimal latency and bandwidth costs, Railway chose ISPs based on geographical network maturity. By consolidating full internet routing tables, the best paths for IP prefixes were determined. Redundancy was ensured by developing multiple zones in each region, using tools like dark fiber for future expansion.
Physical Layout and Installation
Data center racks, arranged for efficient airflow, play a crucial role. Railway chose racks that maximize airflow by aligning cable management with device orientations. Professional assistance was sought for orderly cabling, vital for data and power distribution. Comprehensive documentation, including a cabling matrix and rack elevation, was prepared for installation contractors, detailing connections and device placements.
Construction Challenges
Building a data center is akin to constructing a house, requiring attention to detail. Railway faced challenges such as reverse power feed numbering and non-standard cable wiring. Despite these hurdles, the hardware setup transitioned to familiar tasks like configuring networks and updating internet registry records.
Final Steps and Future Plans
With the infrastructure in place, Railway shifted focus to software configuration, employing tools like Redfish APIs and PXE. Railway’s network design leverages FRR and SONiC on whitebox switches, integrating with the control plane. The journey continues with future development plans to transform server setups into functional Railway zones. Enthusiasts are invited to explore open Infrastructure Engineering roles for a deeper involvement in this evolving endeavor.