History and Future of Cloud Federation

I’m now working for about two years on something called Federated Cloud Sharing. It started on June, 23er 2014 with the release of ownCloud 7.0. Back then it was simply called “Server to Server sharing”. During all this years I never wrote about the broader ideas behind this technology, why we do it, what we achieved and where we are going.

Motivation

The Internet started as a decentralized network, meant to be resilient to disruptions, both due to accidents or malicious activity. This was one of the key factors which made the Internet successful. From the World Wide Web, over IRC, news groups, e-mail to XMPP. Everything was designed as decentralized networks, which is why if you are on the Google servers you can email people at Yahoo. Everybody can set up his own web server, e-mail or chat server and communicate with everyone else. Individuals up to large organisations could easily join the network, participate and build business without barriers. People could experiment with new innovative ideas and nobody had the power to stop them or to slow them down. This was only possible because all underlying technology and protocols were build on both Open Standards and Free Software.

This changed dramatically over the last ten years. Open and inclusive networks were replaced by large centralized services operated by large companies. In order to present yourself or your business in the public it was no longer enough to have your own website, you had to have a page on one or two key platforms. For communication it was no longer enough to have a e-mail address, or be on one of the many IRC or XMPP servers. Instead people expected that you have a account on one of the major communication platforms. This created huge centralized networks, with many problems for privacy, security and self-determination. To talk to everybody, you have to have an account on Facebook, at Google, Skype, Whatsapp, Signal and so on. The centralization also made it quite easy to censor people or manipulate their view by determining the content presented to them. The algorithms behind the Facebook news feed or the “what you missed” in Twitter are very clever — or so we assume, as we don’t know how they work or determine what is important.

The last few years many initiatives started to solve this problem in various ways, for example by developing distributed social networks. I work in the area of liberating people who share and sync all sort of data. We saw the rise of successfully projects such as ownCloud, Pydio and now of course Nextcloud. They all have in common that they built Free Software platforms based to a large extend on Open Standards to allow people to host, edit and share their data without giving up control and privacy. This was a huge step in creating more competition and restoring decentralized structures. But it also had one big drawback. It created many small islands. You could only collaborate with people on the same server, but not with others who run their own server. This leads us to the concept of federated cloud sharing.

Server to Server Sharing

The first version of this ideas was implemented in ownCloud 7.0 as “Server to Server Sharing”. ownCloud already knew the concept of sharing anonymous links with people outside of the server. And, as ownCloud offered both a WebDAV interface and could mount external WebDAV shares, it was possible to manually hook a ownCloud into another ownCloud server. Therefore the first obvious step was to add a “Add to your ownCloud” button to this link shares, allowing people to connect such public links with their cloud by mounting it as a external WebDAV resource.

Federated Cloud Sharing

Server to server sharing already helped a lot to establish some bridges between many small islands created by the ability to self-host your cloud solution. But it was still not the kind of integration people where used to from the large centralized services and it only worked for ownCloud, not across various open source file sync and share solutions.

The next iteration of this concept introduced what we called a “federated cloud ID”, which looks similar to a e-mail address and, like email, refers to a user on a specific server. This ID could then be used in the normal share dialog to share files with people on a different server!

The way servers communicate with each other in order to share a file with a user on a different server was publicly documented with the goal to create a standardized protocol. To further the protocol and to invite others to implement it we started the Open Cloud Mesh project together with GÉANT, an European research collaboration initiative. Today the protocol is already implemented by ownCloud, Pydio and now Nextcloud. This enables people to seamlessly share and collaborate, no matter if everyone is on the same server or if people run their own cloud server based on one of the three supporting servers.

Trusted Servers

In order to make it easier to find people on other servers we introduced the concept of “trusted servers” as one of our last steps. This allows administrator to define other servers they trust. If two servers trust each other they will sync their user lists. This way the share dialogue can auto-complete not only local users but also users on other trusted servers. The administrator can decide to define the lists of trusted servers manually or allow the server to auto add every other server to which at least one federated share was successfully created. This way it is possible to let your cloud server learn about more and more other servers over time, connect with them and increase the network of trusted servers.

Open Challenges: where we’re taking Federated Cloud Sharing

Of course there are still many areas to improve. For example the way you can discover users on different server to share with them, for which we’re working on a global, shared address book solution. Another point is that at the moment this is limited to sharing files. A logical next step would be to extend this to many other areas like address books, calendars and to real-time text, voice and video communication and we are, of course, planning for that. I will write about this in greater detail in on of my next blogs but if you’re interested in getting involved, you are invited to check out what we’re up to on GitHub and of course, you can contact me any time.