Latest interface: 0.3.1
Latest system: 010
Dear ZFSguru users,

It has been a year since my previous 'State of the Project'. With not as much news coming from myself on the forums, I can understand that many of you want an update of what's going on with the project.

While the forums are quiet, ZFSguru lives quietly on the systems of quite a few users out there. It works, it has an easy interface. There are many things to like ZFSguru. But still -- many will feel ZFSguru still is a work-in-progress and far removed from a mature product with a vibrant user community.
Let's see what ZFSguru is doing right:


Easiest ZFS web-interface for novice users, many of whom have only Windows experience
ZFSguru was created as a proof-of-concept that advanced technology like ZFS can also be operated by ordinary human beings, who lack the technical skills and motivation to operate a UNIX system. Apple proved that it is possible to use a UNIX system to provide a user friendly desktop -- Mac OSX is built atop of UNIX. Likewise, ZFSguru proved that ZFS can be utilised by novice users to function as network storage appliance.


Some web-interface pages of ZFSguru are really good and easy to understand without a manual
The aim of the project was to create a web-interface that did not need a manual to be understood. Instead, pages should offer logical functionality that are accessible to novice users, while still being powerful enough to please the more experienced users who want more control. While plenty of improvements can be made in ZFSguru in this regard, overall I would say that ZFSguru has the right touch for web-interface design.


Installation is really easy and comes with few restrictions
ZFSguru is unique in its ability to boot from RAID-Z family pools. It also has very few restrictions in that the boot/system disk can also be used for storage. Additionally the partition map editor allows for flexible use of drives with different sizes, and allows a Solid State Drive to be used as boot disk, cache disk, separate ZIL and overprovisioning. The partition map editor is a visual editor, that should be easy enough to operate and sports neat functions like TRIM erase and partition resizing.


Service addon packages are easy to install and some of them are really good
The mentality of a lightweight 'core' OS with addon packages providing additional functionality, works pretty well. VirtualBox, Transmission, SABnzbd+ and Owncloud are just a few of the well integrated service addons available on ZFSguru. New services are added regularly, though a true invasion of many new and updated services will have to wait until the community can take care of this.


Graphical services allow for optional desktop environment
Another unique feature among ZFS appliances is ZFSguru's ability to run a graphical desktop environment. This can be used for graphical installation, usage of graphical applications like XBMC, even virtualisation with VirtualBox that allows a full-screen Ubuntu Linux session for example. Many things are possible with extended functionality. And with a modular framework it becomes easy to install and use.


ZFSguru supports the newest (experimental) open source graphics stack
The graphics stack is under heavy development to get in line with the quality of the Linux graphics stack. New 'Kernel Mode Setting' KMS-drivers were written for Intel and AMD. A new system console driver called Newcons was also introduced, allowing the use of graphical console together with KMS graphics drivers. Together, it makes ZFSguru the most graphically advanced BSD server appliance.


Advanced users can modify their installation as they please
ZFSguru has never stripped away anything from the base installation, like other NAS operating systems do. Because of this, ZFSguru can be utilised by advanced users to adapt their installation to their personal needs.


ZFSguru system releases are ahead of competition
While other NAS operating systems tend to be more conservative and ship only stable releases, ZFSguru also ships experimental 11.x releases with the latest of operating system development.
That is a nice list of items. But then, one could make a similar list of items where ZFSguru fails to meet user expectations. Some of the items on that list may include:


ZFSguru is still a work-in-progress
Due to the ambitious nature of the project, things are not moving as quickly as some would like. If the project would have been limited to a web-interface only -- such as napp-it -- then it would have been much easier to accelerate the project towards a mature state. However, this would also mean abandoning some key goals of the project. ZFSguru is more than just a web-interface. A project of a more limited scope would not be able to fulfil the needs of the target audience.


Communication with the community about development progress is spotty, at best
While I might be able to produce something useful to other people, communication is not my strong suit. Users like to know what is going on, and regular updates on development progress are desirable to keep users interested with ZFSguru development.


Jason's silly version scheme produced all kinds of beta releases
Because of my adherence to providing 0.2 final as the intended product, I have been postponing a final release for multiple years now. Instead of using numbers to illustrate development progress, I have produced all kinds of beta releases. While some releases are actually quite heavyweight, sporting a lot of new functionality, the general impression of many users is that the development has stagnated.


System images could be more frequent and have more information attached to them
System images are snapshots of the operating system that ZFSguru uses. It contains the kernel, userland and pre-installed applications that provide basic functionality, with add-on services atop of that. While this works out nicely, producing new system images is a daunting task that consumes much of my time and energy. The result is that new releases are sporadic. In addition, not much information is provided about the changes in system releases.


A lot of services are just stubs and only install packages
While adding many services is good, the quality of services varies greatly. Some are really well designed and virtually finished, while others are stubs that only install packages. These stubs should be upgraded to full-blown services with their own web-interface to accommodate the user in configuring the package.


The website and forum lack functionality and look old fashioned
I have been using old scrapes of ugly code to get my website running, in order to prevent security issues that plague major software packages. The result is an old fashioned website and a forum that lacks key functionality - such as the ability to search.


There is no easy way to contribute to the project or even donate money
There is no code repository and collaborated development is impractical. With just two developers, it is not a high-speed locomotive either. As far as donations go, I have been very reluctant in asking for donations in the past. Many people however, have expressed their desire to contribute to ZFSguru. Especially if that helps the project move forward.


While I'm sure I missed plenty in my list, I do think I addressed some very important ones. These are the ones I want to talk about in this State of the Project. :-)
So, what is the idea behind our project? What are we trying to accomplish here?


Where it all began...
To answer these questions, we must go back to the beginning of the project. On the HardOCP forums, I was a strong advocate of ZFS technology and argued that anyone who wanted to provide his or her data with adequate protection, should use ZFS. Other filesystems just don't cut it any longer, due to their total lack of any protection against corruption. Even worse, these legacy filesystems cannot even detect corruption, allowing it to spread to backups. Legacy RAID engines also provide a false sense of protection and may kick drives out of the RAID if just a tiny sector on the harddrive was unreadable.

However, I was fiercely contradicted by several people on the HardOCP forum, who argued that ZFS is way too difficult to operate by non-techies and insisted that Windows combined with Hardware RAID was the way to go. This prompted me to work on a proof-of-concept that provided a minimum to operate ZFS using a web-interface that people could operate, simply their Windows or Mac desktop. Soon enough, I got emerged into the project that ZFSguru is today....


So what *is* ZFSguru?
Soon enough, ZFSguru grew by adding new options such as an installation method and LiveCD to make it even more accessible, completely hiding the command line shell that novice users dislike. With the inclusion of add-on services, the scope of the project was getting bigger and bigger. When ZFSguru was introducing the graphical desktop, people began to ask what ZFSguru really is. A web-interface? A NAS operating system? Some people even got angry, stating that ZFSguru was moving in a direction that they disliked.

In the previous State of the Project, I addressed these questions by saying that ZFSguru is more than just a web-interface to ZFS, more than just a NAS operating system. Instead, I described ZFSguru as a modular server appliance. Modular because of the idea of starting lightweight with only basic functionality present, requiring addons to be installed to provide more functionality. Appliance because it is a tool to be utilised without great effort. Not something you have to study before you can use it. Just like your espresso machine is an appliance; it serves a purpose and -- like many tools -- can be used without much effort or skill.

So, while ZFSguru has its roots as NAS operating system, I have the ambition to extend it towards something even cooler. And all this without betraying its origins. If you just want a NAS, then do not install any addons. Simple as that! You want more, like an email server, webserver, FTP server, cloud storage, torrent and newsgroup client, etc? Then the addons will provide this functionality. This philosophy allows for extended functionality without getting bloated; forcing people to run applications that they do not use nor want.

The flexibility of ZFSguru was demonstrated with the inclusion of a graphical desktop that people can use to configure ZFSguru, to use as virtualisation platform running other operating systems atop of ZFSguru, or running XBMC and transform ZFSguru into a HTPC. I think this is extremely cool! ZFSguru could become something that does not exist yet; a server appliance that is very easy to use and can transform a dull computer to do almost anything you want.

It most likely will not become your favourite desktop, but it might become your ideal server platform that you can deploy with little effort and get things working! Hence the word appliance. Unifying the wide range of interests of our users is a challenge, however. It requires a lot of designing, a lot of redesigning, and a lot of time. That is why ZFSguru is still in an early phase, whereas it could have been in a mature state if the scope of the project remained limited to a ZFS web-interface.


So what *is* the plan?
The plan is to build a foundation on which the project can flourish. This requires a major overhaul of all the infrastructure that is currently in place. Only when the project is on its own feet, I hope to attract some more developers and involve the community in the future of the project. In other words, I need some time to get the locomotive running. But once it is running, I'm rather confident more people will jump the bandwagon.
This year, we have worked pretty hard on all the different areas of ZFSguru. It just isn't very visible to our users at this point. But there is some visible progress:


ZFSguru beta9 web-interface
This year we released ZFSguru beta9 web-interface with advanced drag-and-drop Samba functionality, allowing easy permission handling. It features a whole new Access page with revamped file sharing support. However, not all pages work with Chrome and Safari. Some only work with Firefox. This will be fixed in a future release.


System images
New system images have been released, based on 10-STABLE and 11-CURRENT. Currently, the 10.1-001 is the preferred version to run, since it comes from stable branch. If you have not yet updated, you should be aware that the newly released system version also addresses a high-profile security vulnerability -- known as the SSL heartbleed bug.


Graphics support (KMS/Newcons)
Both 10.1-001 and 11.0-001 system images feature the latest of BSD graphics development. The Kernel Mode Setting (KMS) graphics drivers can offer modern graphics infrastructure that is close to what Linux provides at this moment. In addition to the KMS-drivers, a new console driver called Newcons was introduced. Officially called vt(4) - this replacement allows the use of KMS drivers while still being able to access the console / text-mode. It also introduces Unicode support that can display multi-byte characters.

It also paves the way for cool graphically animated splash screens during boot phase. Many other operating systems have animated splash screens and ZFSguru should stay in league. Support for animated splash is expected to finish this summer.

Sadly, there is no support yet for Intel Haswell (4000-series) processor graphics. Unlike the 2000 and 3000 series, a Haswell-based processor will not be able to use the new KMS-drivers that provide accelerated graphics. Instead, the vga/vesa driver may be used, which works but is very slow and does not support GL-applications such as XBMC. Similarly, there is no support yet for AMD Radeon 8000 and R200 series graphics. So while support of the previous generation of integrated graphics exists, the newest generation is not yet properly supported. Hopefully progress will be made to support these latest generations so that accelerated graphics is possible on ZFSguru for all major hardware.


Services
Some new services have been added. One service called Webstream is exclusively designed for ZFSguru and allows the download of webstreams to the local harddrive as .mp4 file, which can be opened by virtually all media players out there like VLC. It currently only works for the Dutch uitzending-gemist website, but other webstream providers can be added in the future.
While both CiPHER and I have discussed pretty ambitious plans for the future, there are some items that are already being worked at. You can expect to hear more about these items in the future:


Advanced build infrastructure
The current method of producing new system images still involves many human steps. The secure the future of the project, having an infrastructure that automates the building and release of new system images is crucial. Much of my time in the past months has been spent on the creation of a new build infrastructure that can build and release new system images. Once finished, it should free my hands of a major chore and allow me to spend time on different areas of the project.


Mesa modular framework
While moved somewhat to the background, the Mesa project aims to build a framework that ZFSguru can hook on to provide a high quality toolkit/library solution, as well as provide the basis of a content management system that can provide the basis of a new website. Mesa is in a working state, but needs some core changes before I can utilise it. Integration with ZFSguru is something I expect can happen next year. This will provide ZFSguru with more mature library functions that are easier to maintain, particularly when the project starts to attract more developers.


New website, new forum
A new website is being worked on. The design is not yet complete, and requires Mesa to be enhanced further to the point where it can be used in production. Additionally, the forum rewrite 'newforum' is still at an early stage. I wanted to have the new website online pretty soon, but I guess it will take a little bit longer. The new website should be very interesting because it allows me to integrate ZFSguru activities on the main website. What that means? Just wait and see. :-)


GitLab development website
To allow for collaborated development, it must be easy to report bugs and submit code. To this end, developer CiPHER has immersed himself into GitLab, a project inspired on GitHub that serves as a drop-in replacement. Developer CiPHER has created a service addon for GitLab, so that any ZFSguru machine can run this development framework. It currently still has some issues, but once the new Alpha server is online we hope to use GitLab as development platform.


Server infrastructure: new Alpha master server
The most important part of the ZFSguru server infrastructure is the Alpha master server. Currently I'm paying for a USA-based server. But this server is virtual, so we do not have access to the hardware. For security reasons, it would be preferable to have a highly secured Alpha master server.

Developer CiPHER has taken upon himself to act as server admin for the new Alpha master server. This is the prime server that has the most authority over the ZFSguru infrastructure, both now and in the future. If compromised, the alpha master server could feed virus-infected services and other downloads to our users, which we want to avoid. The new server will be security hardened. Because we have access to the real hardware, we can provide some real security on a software level, and adequate protection against unauthorised hardware access.

The server will be located in Amsterdam, The Netherlands. This European country has excellent peering connections as it is a major hub for internet connections worldwide. This should provide low latencies to a number of countries, making it very suitable for a ZFSguru master server. With developer CiPHER being located in The Netherlands, physical maintenance is also taken care of. The server will run ZFSguru itself, though configured in a way to provide more security. Eating your own dogfood, or something like that. :-)


Web-interface
It is somewhat ironic that the web-interface hasn't gotten much love lately. While beta9 was released early this year sporting some nice functionality, the plans for beta10 have slipped somewhat.

Beta10 will have some minor bug fixes, updated libraries and will sport new functionality. It will introduce a new remote database architecture dubbed 'GuruDB' which uses compression and intelligent caching. This should speed up some pages of the web-interface when accessed after a period of inactivity.

But the real cool stuff is the new Bulletin feature, where you can read ZFSguru Service Bulletins. There are messages from the developers to warn users about new releases, changes in functionality, security vulnerabilities or other important information. When I release a version that has important errata, this is the way I want to communicate to my users. For example, the need to update bootcode when using ZFS version 5000 would be a perfect candidate. In the meantime I opted to present this information on the Download page, but this is far from ideal since existing users might not see this information at all. A good way for us to communicate with our users is pretty much mandatory, if the project wants to advance its success. The new Bulletin feature is powered by the GuruDB database.

Looking further ahead, Beta11 and successors will deliver the long awaited features that blocks a first stable release. One of these key features still missing is the almighty Migration Manager. This functionality is essential to the way ZFSguru wants to provide system updates to users, as it allows users to upgrade the operating system without losing their settings.

Work on features beyond 0.2 have already commenced. In particular, providing more functionality on the Network page is something that has already shown some progression. Other features are still being explored. I am still in doubt about the order of development. Maybe the community should be involved in this decision, since there are multiple roads that lead towards the goals envisioned by developer CiPHER and myself.
Please allow me to conclude that I understand not all of our users' needs are being met at this early junction. However, we are working hard to make ZFSguru a really cool project. It is the potential of the project and the many ideas that give us the energy to keep working on it. Once finished, it is going to be something really cool!

Thank you for your patience and understanding.

Kind regards,
Jason - developer and project architect
CiPHER - developer and server administrator

Valid XHTML 1.1