Latest interface: 0.3.1
Latest system: 010
karmantyu
User

157 posts

Posted on 3 February 2018 @ 19:04edited 2 March 2018 @ 07:58
The time has come. The latest ZFSguru distro became obsolete. If you dare to make a # pkg upgrade you will most likely break some important functions like SAMBA.
If you would like to use the nice ZFSguru webinterface and have an updated system there is a quite easy way to do it I show you how.
Do not forget to do this upgrade - if you wish - till April 2018 (or save the posts important for you). ZFSguru.com domain may go down soon.

At first here is something to read:
https://www.freebsd.org/doc/handbook/updating-upgrading-freebsdupdate.html
And read at least to STEP 3 please.

STEP 1
freebsd-update fetch install # at first try to patch your existing system to the latest patch level
freebsd-update -r 11.1-RELEASE upgrade # make the upgrade

# ZFSguru is running on a nongeneric kernel so we need to get a generic kernel.

cd /tmp/
fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.1-RELEASE/kernel.txz
mv /boot/kernel/* /boot/kernel.old # save the ZFSguru kernel to the kernel.old directory
tar -C / -xzvf kernel.txz # install vanilla kernel to its place.
shutdown -r now # restart

freebsd-update install
shutdown -r now

STEP 2
# You may need again:
freebsd-update install
shutdown -r now

# Now we should do a forced upgrade of all installed packages. They will be replaced by the packages with fresh versions from the repository even if the version number has not increased. This is required because of the ABI version change when upgrading between major versions of FreeBSD. The forced upgrade can be accomplished by performing:

pkg-static upgrade -f

# Once the software upgrades are complete, finish the upgrade process with a final call to freebsd-update in order to tie up all the loose ends in the upgrade process:

freebsd-update install

STEP 3
# Final things to do: During portmerge there will be surely some configuration file changes which has to be sorted out. The merging process may write in the /etc/hosts.conf; /etc/sysctl.conf; /usr/local/etc/smb4.conf; /usr/local/etc/ssh/ssh_config; /usr/local/etc/ssh/sshd_config and to many other places. If you do not edit out the >>>>>>>>>>>>>> and <<<<<<<<<<<< comments from the config files, your server may not boot up or server functions will be degraded. So take a note about the merged config files during the merge process - you will be noted about problems - and edit the files before rebooting.

# Cleanup a little bit:
pkg autoremove
pkg clean

STEP 4
# You need this step only if your server has some extra hardware or for any other reason you may need a custom kernel. Let see how-to to compile one for yourself. ************************************

cd /tmp/
fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.1-RELEASE/src.txz # get the source
tar -C / -xzvf src.txz # copy it to its place
cd /usr/src/sys/amd64/conf # Please choose the directory reflecting your CPU architecture e.g. amd64 here.
cp GENERIC MYKERNEL # copying the default kernel configuration file to filename to edit
mkdir /boot/MYKERNEL # creating destination directory for the new kernel

########### Here you can make config your kernel if you like or need to. Let's see an example.
ee MYKERNEL # open new kernel configuration file to edit
# Remove unnecessary driver options and change ident to your liking.
ident MYKERNEL # Will be our kernel name.
options DEVICE_POLLING # Give the option to the kernel to poll devices like network controllers. It helps to keep week/old systems under heavy load to stay responsive.
nomakeoptions DEBUG # I don't need to debug my server
#makeoptions DEBUG=-g # uncomment &#8211; not needed if no debugging used
#makeoptions WITH_CTF=1 # uncomment &#8211; not needed if no debugging used

# After saving MYKERNEL configfile edit the src.conf file options.
ee /etc/src.conf
WITHOUT_DEBUG_FILES=1 # no debugging needed
WITHOUT_KERNEL_SYMBOLS=yes # so no symbols needed
MALLOC_PRODUCTION=YES # also unnecessary then

# After saving src.conf file edit rc.conf
ee /etc/rc.conf
ifconfig bge0 polling #add this line and substitute the network adapter name with your servers if the network traffic requires it.
# see https://networking.ringofsaturn.com/Unix/freebsd-polling.php

# After saving rc.conf edit loader.conf for options.
ee /boot/loader.conf
hw.usb.xhci.use_polling="1" # for example

# Save loader.conf. We have every configuration ready so let's build kernel.
cd /usr/src/
env DESTDIR=/boot/MYKERNEL make kernel KERNCONF=MYKERNEL
mv /boot/MYKERNEL/boot/kernel/* /boot/MYKERNEL # copy kernel from build directory
rm -rf /boot/MYKERNEL/boot # remove not needed dir
nextboot -k MYKERNEL # specify kernel to boot from
shutdown -r now # reboot

I assume everything went well and your server booted up and running, so you can copy the kernel to its place.

mv /boot/kernel/* /boot/kernel.old # save old kernel
cp /boot/MYKERNEL/* /boot/kernel # make new kernel permanent
shutdown -r now # reboot


You may need to make some adjustments to the web interface too. Please take care to read the next post:
https://zfsguru.com/forum/zfsgurusupport/1165

Your server will use SAMBA 4 from now on. If you need SAMBA please read the post here:
http://zfsguru.com/forum/zfsgurusupport/1149

If you would like to have VNC- you have some help in this post here:
http://zfsguru.com/forum/zfsgurusupport/1120#6883

Take care and may the force be with you :) V
hgeorges
User

10 posts

Posted on 11 February 2018 @ 15:13
Hi karmantyu;
Thank you for posting these upgrade notes.
I have had my system for quite a few years - mostly for backup, and had little to interact with. Decided to upgrade (11.1.008) following them, and I guess did it too rushed. Ended up with an unbootable system (you've certainly warned about this possibility). Since I had a previously saved boot image (11.1.009), I was successful activating and booting that one using the LiveCD. So I'm "back in business" - only "one life" less :)
If you don't mind a couple of questions - hopefully I can do this one more time successfully...
Starting with the first step:
freebsd-update -r 11.1-RELEASE upgrade
this does a lot of checking, downloading updates etc. It complains that the machine is in fact not upgradable due to non-standard kernel, and yet, somehow it moves ahead with the downloads and then walks through the 4-5 config files which are different.
I've essentially removed the old sections, and retained the newly inserted headers. Removed the >>> or <<< sections - left the respective files clean.
There was nothing to adjust in settings as far as I saw.
At the end of each file I just exited and saved, and the flow moved me to the next. When finished, I was again at the prompt, but the procedure displayed a script for me to run, from /usr/... which I did, then I rebooted. Upon rebooting I ended up with invalid disk config error, been given a prompt to do something about that situation, and that's where I was stuck. Was I supposed to execute that script? I suppose I should. Was I supposed to reboot after, or move to your next step and fetched the kernel? What is here the correct sequence of operations which would render a bootable configuration?
Are any of the kernel features ZFSGuru has essential to be enabled somewhere (before booting the GENERIC kernel)?
Please shed a bit more light on the steps which need to be executed.

Essentially I have only one more try with it as is, and I'd like to keep the web interface and everything, but if I use that and miss, I'll probably have to move to N4F, or FN, import my pools there and abandon the zfsguru interface and convenience.

Thank you and excuse the long read.
hg

karmantyu
User

157 posts

Posted on 12 February 2018 @ 09:52edited 10:32 29s
Hi hgeorges,

If I understand well you made the upgrade and rebooted without changing the kernel to GENERIC. In STEP 1 you had to change to the new kernel before rebooting. ZFSguru is not requiring any special kernel features enabled it will run on GENERIC kernel perfectly. First reboot should happen first in STEP 2 after upgrading kernel!

I think there is nothing to worry about yet. You can always make a clean install from a downloaded FreeBSD iso, set up the ZFSguru WEB interface and import the pools. Hopefully you do not have any extra ZFS pool extension features enabled.
http://zfsguru.com/forum/generalchat/946
http://zfsguru.com/doc/bsd/zfsguru

As far as I don't see your boot process I can't tell you what options exactly you have. So at first be sure your bootdisks bootcode is up to date. If you can boot up from boot image (11.1.009) then you have ZFSguru web interface and in the DISK menu there is the option to update the ZFS bootcodes. After doing this you may try again to boot up again to see how far you get with the old kernel. You may have boot menu and some limited environment to work in. If that's the case you can upgrade to the GENERIC kernel and boot up successful. I'm not aware of any script have to run after upgrade. Maybe FreeBSD generating one if the system is older than a specified version or it was simply necessary to run to upgrade your bootdisks bootcode.

Wish you luck,
arr bee
User

13 posts

Posted on 12 February 2018 @ 12:55
Apologies if I have this a bit wrong, But it looks to me like step 1A acquires the generic kernel but it is not used until step 2?

#cp /boot/GENERIC/* /boot/kernel/

Step 2 has a reboot prior to this step. could this be the problem?
karmantyu
User

157 posts

Posted on 12 February 2018 @ 18:28edited 18:38 48s
It seems to me you have trouble too. I don't know what could have gone wrong. Please describe your problem more exactly.

STEP 2 begins with reboot so you don't use the new kernel until then.

STEP 1/A copying the new kernel directly in its place but it will kick in first by rebooting in STEP 2.

STEP 1/B compiling a new kernel from source (which is the preferred method for me) and by the first reboot in STEP 2 the kernel will be loaded from the /boot/GENERIC directory. After successful reboot we can be sure the kernel is working and copying it to its rightful place /boot/kernel/.

So I made some small changes to the original post to be clearer.
arr bee
User

13 posts

Posted on 15 February 2018 @ 14:47
I didn't have a problem as I hadn't attempted this procedure yet.
I was just trying to identify what *might* have been an issue for hgeorges but I see that I got it wrong. :)

I am running through this myself in a VM though (as a POC) and have a couple of comments/questions:
1. in step2 do we need to take the sub directories in the copy command after compiling the kernel in step 1B?
eg should it be #cp -R ....

2. When doing a "freebsd-update install" I sometimes get the response "run a fetch 1st". I assume I should run the fetch if I get this or can it be ignored?
[root@zfsguru ~]# freebsd-update install
No updates are available to install.
Run '/usr/sbin/freebsd-update fetch' first."

3. when running the fetch An integrity check fails.
[root@zfsguru ~]# /usr/sbin/freebsd-update fetch
Looking up update.FreeBSD.org mirrors... none found.
Fetching metadata signature for 11.1-RELEASE from update.FreeBSD.org... done.
Fetching metadata index... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Fetching 2 metadata files... done.
The update metadata is correctly signed, but
failed an integrity check.
Cowardly refusing to proceed any further.

This seems to be caused by upgrade path incompatibility and was solved for me by starting the whole process with patching to the latest level on my from state:
#freebsd-update install

4. when running the nextboot command as part of 1B I got a message about something being readonly and so the loader wouldnt be updated. (scroll back buffer was too small for me to find the message now)
I checked /boot/nextboot.conf and it was updated so I proceeded and the system came up OK.


5. when doing the pkg upgrade in step3 there are a lot of "pkg-static: Skipping unknown key 'messages'" messages.
is that an indication of some missconfiguration?
it does go on to update various pkgs

6. also during the pkg update there is an error with the _tss user which causes an exit.
If I continue at this point without fixing the web interface breaks.
the fix is to refresh the pw database (just used vipw to save the file)

ultimately my attempts fail.
The process seems to work, but after doing the pkg-static upgrade -f, any reboot results in no web interface.

I don't have enough knowledge to troubleshoot further.




Cheers,
RB
arr bee
User

13 posts

Posted on 15 February 2018 @ 15:17
What I would be interested to know is....

The install section of the webUI has many versions listed but I have only been able to download 10.2.003. any other that Ive tried seems to have vanised (as will happen when a project winds down I guess).
Does anyone have any later liveCDs stored somewhere that I might be able to copy?
Maybe a 11.0.10 or 11.1.10??

Cheers,
RB
karmantyu
User

157 posts

Posted on 16 February 2018 @ 10:39edited 13:44 39s
arr bee wrote: I am running through this myself in a VM though (as a POC) and have a couple of comments/questions:

1. You can use of course #cp -R **it is not needed though
2. You should do #freebsd-update fetch install **you may have nothing new to update to
3. I forgot about the fact if you want to upgrade to 11.1 from 9.x or 10.x you need to make sure you're running the fixed version first (9.3-RELEASE-p45, 10.1-RELEASE-p37, 10.2-RELEASE-p20 or 10.3-RELEASE-p6). So you handled the situation well with # freebsd-update fetch install -at first.
4. I'm not sure what the system may found write protected.
5. I'm not sure again - are you sure the system is running on a fresh compiled GENERIC kernel?
6. Maybe your system config was too old or too specific. You may need a clean and fresh install.

Web-interface is still here http://alpha.zfsguru.com/interface/ZFSguru-0.3.1-webinterface.txz

I'm not aware of 11.1 LiveCD releases. Latest you can have is 10.2.003 with web-interface 0.3.1.
hgeorges
User

10 posts

Posted on 17 February 2018 @ 03:07
Hi All, and thank you for continuing to clarify this topic. I've been extremely busy and have not had the time to go back to my planned upgrade. Hence my lack of progress.
Since I only have one bootable image (I messed up the other one) I need to dedicate my full attention to the procedure, so that I don't mess up anything.
My complication resides with the fact that I relied on a couple of SSD's for zfs root (mirrored), swap, l2arc and zil partitions. I want to keep things as they are since they work fine - hence my caution.

One thing I did though, I've loaded Nas4Free on an usb stick, and after booting from it, I've been able to import my pools, all partitions have been recognized and made sure I can access my data.

I have therefore a plan B, if this upgrade doesn't render ZFSguru usable.
karmantyu
User

157 posts

Posted on 17 February 2018 @ 15:09
Good :)
Qui
User

60 posts

Posted on 18 February 2018 @ 09:38
thank you so much for this karmantyu :)

worked like a charm, I had one problem with perl when updating the packages before doing the last freebsd-update install, then it all went through without a hitch.

These were the steps I took. (reediting your lines)


freebsd-update fetch install
freebsd-update -r 11.1-RELEASE upgrade
# here it asked me to run the /usr/sbin/freebsd-update install
# but I waited and first fixed the Generic kernel with step A.

##
I actually had to run this part somewhere here to fix <<<<<<<< >>>>>>>>> config stuff
--
During portmerge there will be surely some configuration file changes which has to be sorted out.
The merging process may write in the /etc/hosts.conf; /etc/sysctl.conf; /usr/local/etc/smb4.conf; /usr/local/etc/ssh/ssh_config; /usr/local/etc/ssh/sshd_config and to many other places. If you do not edit out the >>>>>>>>>>>>>> and <<<<<<<<<<<< comments from the config files, your server may not boot up or server functions will be degraded. So take a note about the merged config files during the merge process - you will be noted about problems - and edit the files before rebooting.
##

ZFSguru is running on a nongeneric kernel so we need to get a generic kernel.

A) The easy - standard way: (Less preferred easy way contains more risks.)

cd /tmp/
fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.1-RELEASE/kernel.txz
tar -C / -xzvf kernel.txz

after this i ran the:
/usr/sbin/freebsd-update install

and rebooted


Then I fixed packages and ran install again afterwards just like it says in your lines:

Now we should do a forced upgrade of all installed packages. They will be replaced by the packages with fresh versions from the repository even if the version number has not increased. This is required because of the ABI version change when upgrading between major versions of FreeBSD. The forced upgrade can be accomplished by performing:

pkg-static upgrade -f

Once the software upgrades are complete, finish the upgrade process with a final call to freebsd-update in order to tie up all the loose ends in the upgrade process:
freebsd-update install

finished up with:

Cleanup a little bit:
pkg autoremove
pkg clean
arr bee
User

13 posts

Posted on 19 February 2018 @ 12:09
karmantyu wrote: 1. You can use of course #cp -R **it is not needed though
2. You should do #freebsd-update fetch install **you may have nothing new to update to
3. I forgot about the fact if you want to upgrade to 11.1 from 9.x or 10.x you need to make sure you're running the fixed version first (9.3-RELEASE-p45, 10.1-RELEASE-p37, 10.2-RELEASE-p20 or 10.3-RELEASE-p6). So you handled the situation well with # freebsd-update fetch install -at first.
4. I'm not sure what the system may found write protected.
5. I'm not sure again - are you sure the system is running on a fresh compiled GENERIC kernel?
6. Maybe your system config was too old or too specific. You may need a clean and fresh install.

Web-interface is still here http://alpha.zfsguru.com/interface/ZFSguru-0.3.1-webinterface.txz

I'm not aware of 11.1 LiveCD releases. Latest you can have is 10.2.003 with web-interface 0.3.1.


In my case it was a fresh-fresh install as I was just testing in a VM. There wasn't even any user config like datapools, users, extra services etc.
I'll keep plugging away at it and may even try a fresh 11.1 install and add the web interface...
So any extra findings would be appreciated in this thread. I'll keep an eye on it!

Thanks.
karmantyu
User

157 posts

Posted on 19 February 2018 @ 18:11
I'm happy everything went well.
arr bee
User

13 posts

Posted on 21 February 2018 @ 16:58edited 17:01 00s
Ahhhh, Bugger me.
I see where I went wrong now!!

I had failed to finish up with http://zfsguru.com/forum/zfsgurusupport/1165

I thought it was odd that the server can up OK and "looked OK" but the webUI wasn't accessable. :) Oh well, I've got mine working and now have the confidence to do it for real on my server.

Thank you karmantyu!
and Qui. :)

There is one more question I have.
In Qui's method there is a slight difference, and I'm not sure if it was important (and needs to go into the original post).

Qui says:
when using method 1A, update kernel to generic, followed by the freebsd-update install (as requested by screen output when doing the freebsd-update -r 11.1-RELEASE upgrade.
This seems to perform other updates to the kernel??

But the original instructions say to do a reboot immediately after making the kernel generic and then do the freebsd-update install after the reboot.

Any difference doing it either of those 2 ways?
arr bee
User

13 posts

Posted on 21 February 2018 @ 19:11
Additional note:

Samba was upgraded to 4 and wouldn't start.
I seem to have it working now by:
-update rc.conf with "samba_server"
-copy smb.conf to smb4.conf
-update the deprecated "security" value from "share" to something else (I chose "user")
-update /usr/local/www/zfsguru/pages/access/samba_settings.page to comment out "Share-level share" option so it's not accidentally used on future shares

I think thats all I did.
karmantyu
User

157 posts

Posted on 21 February 2018 @ 20:28
arr bee wrote: Additional note:

Samba was upgraded to 4 and wouldn't start.
I seem to have it working now by:
-update rc.conf with "samba_server"
-copy smb.conf to smb4.conf
-update the deprecated "security" value from "share" to something else (I chose "user")
-update /usr/local/www/zfsguru/pages/access/samba_settings.page to comment out "Share-level share" option so it's not accidentally used on future shares

I think thats all I did.

It's quite nice to hear it worked out finally.
For SAMBA4 see http://zfsguru.com/forum/zfsgurusupport/1149
karmantyu
User

157 posts

Posted on 1 March 2018 @ 11:24
Refreshed instructions.
Last Page

Valid XHTML 1.1