Rendered at 02:41:24 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
internet2000 23 hours ago [-]
Starting with robocopy is nuts. I'd have started with rsync and just let it run overnight. The peace of mind that it'd do the right thing and not needing to research flags are more than worth any speed tradeoffs.
Oxodao 19 hours ago [-]
I recently migrated from a synology to a diy nas, 8to to transfer, did it through rsync, took a few days but it ended up working nice.
JoeBOFH 16 hours ago [-]
I understand where you might be coming from but: robocopy SOURCEPATH DESTPATH /mir /r:1
Is pretty easy to remember.
walrus01 18 hours ago [-]
Given Ubiquiti's history of abandoning entire product lines, I cannot ever imagine trusting a ubiquiti closed-source NAS product with my data. No thank you. Not when TrueNAS exists.
simonjgreen 16 hours ago [-]
Basing your major infrastructure choices on Ubiquiti feels a kickstarter purchase every time. Huge claims, borderline religious following community, an intentional and almost proud absence of technical support, and a coin toss if the product line will still exist in 24 months.
And when they work as advertised they are fantastic.
donatj 17 hours ago [-]
> Given Ubiquiti's history of abandoning entire product lines
Huh, this is news to me but seems to check out from my Googling. I've really been enjoying the Ubiquiti ecosystem and was considering a UNAS to replace my Seagate NAS.
walrus01 17 hours ago [-]
I've been following ubiquiti since the days of 802.11a based radio bridges (and then 802.11n based stuff like the powerbridge m5), it's an exceedingly frustrating company to rely on doing anything serious with.
ksec 11 hours ago [-]
802.11a is from 2000. That is 26 years ago! Has there been any major issues with them in the past 5 years?
walrus01 11 hours ago [-]
What I meant was, that prior to about 2012, they put a solid and honest effort into building useful networking equipment for people. Then after that ubnt pivoted to mass market consumer goods. The 802.11a and n stuff was solid.
In the past 5 to 7 years they have abandoned a number of entire product lines and left the people running it hanging, with no way to purchase replacement parts, or keep a system running.
ShinyLeftPad 12 hours ago [-]
I recommend to check about UNAS speed issues, I heard it's pretty slow due to bad controller and on Macs has extra disconnect/time machine issues
bayindirh 17 hours ago [-]
Yes, TrueNAS works great sans a couple of quirks with external storage boxen. I'm mostly a happy camper.
I believe I'll be more happierer as I integrate it to my life more and more.
throwaway270925 19 hours ago [-]
I agree with Scott, no need to use rsync - he should have used rclone instead! It can be easily installed in the package manager, and with 4 threads (or multiples of 4) it fully uses/saturates the 4 1GbE links.
Also a hint for anyone planning something similar: if you use Synologys SHR format (based on mdraid+btrfs) and your new destination supports or uses BTRFS too, you can just btrfs send your whole pool to the new NAS!
(Since its a single process it wont saturate any LAG though)
sersi 16 hours ago [-]
I tried that before and ran into issues due to the fact that synology's btrfs implementation has quite a few patches. I forgot exactly what issues I had, just ymmv.
The fact that synology's btrfs is incompatible with a normal linux distribution and that the official instructions from synology for reading it is to use an older kernel together with ubuntu 16.04 lts convinced me to switch away from synology.
Neywiny 24 hours ago [-]
Sometimes when we Linux for a long time, we run into decades old friends: grep, sh, bc, whatever. Not uncommon to run a command that first released 50 years ago. Rarely on Windows do we get that. As much hate as windows gets, at least they left in more than just explorer drag and drop.
If the author is reading, it would be nice to clarify that the 10g network comment is irrelevant. I was under the impression at least 1 of the devices would have a 10g link, but at least 2 of the 3 didn't. It was a bit confusing, but it's also very late where I am.
joshka 23 hours ago [-]
My interpretation is that the setup is 10GbE UNAS Pro 8, 4x 1GbE Synology DS 1520, 1x 2.5GbE Windows host.
ZPrimed 22 hours ago [-]
If that's the setup, the only way anything would run more than 1Gb is if the Synology is doing SMB-multichannel (which if they setup 4x1Gb LAG, then it's not SMB-MC).
[edit] reading the article, they do have the Syno setup for SMB-MC and that's the only reason this wasn't stuck at ~120MB/s
kalleboo 23 hours ago [-]
The Synology web UI lets you mount a remote SMB share and copy files directly in their file manager without making the round-trip via your PC, I imagine this would have been the fastest method without getting bogged down in command-line tools.
teekert 22 hours ago [-]
I didn't read the whole thing, but couldn't you just rsync over ssh from either end?
Your suggestions is also very nice, you can just turn off your own PC and wait for it to finish.
kalleboo 21 hours ago [-]
The situation in the article is one NAS is 10 Gbit and one NAS had 4x1Gbit. SMB will use MultiChannel to aggregate the 4x1GBit ports, but rsync will just pick one and be limited at 25% of the available bandwidth.
I don't even think rclone can split across multiple interfaces like SMB does.
ZPrimed 22 hours ago [-]
Synology actually supports the old (non-encrypted) rsync protocol, too. On a trusted LAN, it runs much quicker that way than with rsync-over-ssh...
I just finished slurping all of my files off an old Synology and onto a TrueNAS server. Unlike the article here, my Syno is 2x1Gb in a LAG, so I was really only getting ~1Gbps at max.
One disappointment with TrueNAS is that it doesn't support SMB-MC unless you put each interface on a separate subnet (which is a strange limitation because Samba itself doesn't seem to suffer from this, since synology handles it fine)
spockz 20 hours ago [-]
What is the benefit of SMB-LC over managing the link speed at a lower level via link bonding or trunking?
kalleboo 19 hours ago [-]
Most methods of link aggregation won't speed up a single TCP connection, and they can be messy-to-impossible to set up on non-Linux machines.
With SMB you just plug in 2 NICs and you automatically get ~double the bandwidth.
amaccuish 18 hours ago [-]
I tried that once but the cifs kernel mod was so old it didnt support modern auth methods.
c0nsumer 24 hours ago [-]
I love this kind of step-by-step solving post, but it bothers me that the author didn't track down what the JPEG in the Alternate Data Stream (ADS) was for. Something put it there... Might it be a surprise when it ends up not being there later?
Also, it kinda sounds like the UNAS might not be supporting ADS' for some reason? That alone could be a problem later on...
joshka 23 hours ago [-]
Fairly reasonable explanation is album cover art.
c0nsumer 12 hours ago [-]
For sure, that's what I thought about as well. But it also made me think "hang on, this migration will throw out data... Would I be okay with that?".
justsomehnguy 18 hours ago [-]
> Also, it kinda sounds like the UNAS might not be supporting ADS' for some reason? That alone could be a problem later on...
The whole reason of the post.
And the thing what linux fanbois with rsync/rclone here absolutely ignore.
ksec 11 hours ago [-]
Does anyone know if the UNAS range uses ZFS as well or does it use BTRFS or something else?
I only know their higher end enterprise range specifically said ZFS but all the others doesn't seems to mention it. And I really wish there is a simple 4 bay ZFS NAS.
bananamogul 24 hours ago [-]
"Ideally you'll find more value in the comments as Hacker News folks and Windows experts will drop in with better tools and strategies."
I don't know if this is better, but...an alternative is rclone, which uses rsync under the hood. It can also do parallel jobs and is restartable on any failed partial copy.
When I've had to do large Synology-to-Synology moves, that's what I've used. The multiple rsyncs in parallel really helps if you've got a lot of small files.
randerson 23 hours ago [-]
Moving Synology-to-Synology, I was surprised at how easily I could just move the original HDDs (in their original order) from old enclosure to new enclosure, and it just worked.
UltraSane 23 hours ago [-]
I consider it much better because rsync can detect bit errors and robocopy can't.
tamimio 23 hours ago [-]
The issue with rclone compared to rsync for existing files is it will copy the whole thing again, where rsync uses something called delta transfer and only transfer the change. But if you are doing one time cloud to cloud transfer then yeah rclone is the gold standard.
> rsync under the hood
Is it? I think one is written in C (rsync) and the other is in Go lang (rclone)
jaden 23 hours ago [-]
I had the same reaction. rclone is not built on rsync. It's an independent implementation in Go, inspired by rsync.
tamimio 21 hours ago [-]
Mostly the confusion is because they share similar command line interface like —exclude etc
8fingerlouie 18 hours ago [-]
I'm aware of the comments saying rsync or rclone, and I absolutely also use those, but for initial migrations I often find myself simply using tar instead.
Something like :
(cd /src/dir && tar cf -) | ssh user@host "(cd /dst/dir && tar xvf -)"
will do nicely.
It won't saturate 4 multichannel links, you'll most likely need rclone or rsync for that (or robocopy it appears), but it's much faster than rsync on initial copies.
Then for good measure, once the tar copy finishes, I run a single rsync verification between the two targets.
That being said, my latest storage migration was performed using either ChronoSync or Carbon Copy Cloner from my Mac mini server. They won't do anything rsync cannot already do, except present data and copy history in a graphical environment.
somat 16 hours ago [-]
For large transfers on a network I control I would use bare netcat, much faster than going over ssh and without having to figure out rsync server (probably easy, but I don't need another distraction.
nc -l 8888 | tar -C /dir -xzf -
tar -C /dir -czf - | nc server 8888
rsync however, is totally worth it if there is any chance the transfer gets interrupted.
jasode 17 hours ago [-]
>simply using tar instead
Be aware that tar doesn't preserve creation/birth times and keeping that original metadata is important to some people if the target file system can record it. Robocopy preserves/transfers creation times by default and recent rsync versions (3.2.3 2020-08-06) can do it with the extra option --crtimes.
abcd_f 20 hours ago [-]
He should've given bvckup2 a try.
Asynchronous IO, multithreaded for smaller files, serialized when they are big, block-level delta copying on subsequent runs. Pretty much as fast as it gets on Windows.
mmetzger 23 hours ago [-]
The moment the /Z option showed up I figured there'd be an issue. I have to copy multi-TB files often using Robocopy. We originally used /Z because "Well, if it gets interrupted, I'd obviously want to restart from where I left off..." I accidentally left the switch off one day and noted it completed in less than half the normal time (of using the /Z flag).
In other words, yes, it may be restartable, but if it takes more than twice as long, just recopy the file(s) if needed.
para_parolu 23 hours ago [-]
Has anyone found a solution to lack of encryption in unas?
gregoryl 23 hours ago [-]
It has encryption.
tiew9Vii 22 hours ago [-]
From the benchmarks I’ve seen there’s a significant throughput degradation with encryption enabled on the UNAS’s due to limited cpu.
mmastrac 24 hours ago [-]
rsync tends to be a bit tougher on smaller CPUs. scp is generally better for those cases if you don't mind hashing afterwards to confirm everything made it without bit flips (in theory it shouldn't happen, but I've definitely had corruption during transfers like these). I personally prefer that method because you check files that are out of page cache and at rest on the filesystem.
If you want to get the advantage of SMB multi stream copies (in rsync*), look for LACP or other options. Not sure if Synology does that though.
I'm honestly surprised that SMB was faster here. I've never had luck getting it to perform well, so this is a bit of a shock.
luma 23 hours ago [-]
LACP does not work well with SMB multipath and will generally prevent you from seeing the benefit. You want IP addressed adapters for each link on both sides of the conversation.
sirjaz 24 hours ago [-]
It shouldn't be shocking SMB is very fast and more secure than NFS. Hell it supports running over quic now
tuananh 24 hours ago [-]
i recently went from synology to truenas all flash. i do not miss synology one bit. the opensource ecosystem is already there.
for example, Immich is a huge upgrade to Synology photos. I didn't know how much i was missing out.
UltraSane 23 hours ago [-]
Flash is rather expensive for storing photos and video.
tuananh 22 hours ago [-]
yeah right but that's the trade-off i made for total silence & speed :)
UltraSane 21 hours ago [-]
you can get 5 to 10 times more GB/$ with magnetic drives. The best solution is something that is smart enough to stream linear reads like movies from disk but cache data with random IO patterns on flash.
tuananh 20 hours ago [-]
i'm aware of it. i waas using syno with flash cache before. to each their own i guess
9 hours ago [-]
joshka 23 hours ago [-]
It really feels like rsync should have a multi-path setting that would have sped this up across multiple links (but doesn't because noone has made it yet).
jbird99 23 hours ago [-]
I use bbcp for that, it supports multiple streams.
You could just run multiple rsync processes using different src and dest paths to achieve multiple copy streams. Anyway this would not be likely much faster if rsync is run over encrypted SSH due to CPU performance, compared to unencrypted SMB.
joshka 2 hours ago [-]
I took a look into it mostly to satisfy my curiosity. The AES-NI chip on the J4125 in the Synology DS1520 hits 5Gbit/s transfer when using 4 cores (AES-256-GCM). So technically this it would be possible to saturate its 4 ethernet ports (or at least the encryption wouldn't be the bottleneck). You could back this down to 128 or drop the encryption altogether on an network that you own like this reasonably.
My guess is the actual bottleneck would be directory traversal and metadata stuff - i.e. trying to keep the pipe full, not saturation effects.
Yeah I know on src/dest. I've done this sort of approach countless times in various technologies in the past 40 or so years, but that always puts the mechanism for checking things in onus of the human rather than having that one command that does it all right.
Anyway, not a slight on rsync in the slightest - it doesn't have this probably because no-one realistically actually needs it most of the time.
TiredOfLife 22 hours ago [-]
If both devices are capable of running Syncthing, I would just use that.
npodbielski 8 hours ago [-]
I bought several cameras and they unvr early this year. It works great but I have no AI features because you have to buy they hardware just to accept some license (!). I mean... OK but no.
If they would allow me to send them email saying: "I hereby promise I wont record my neighbours having little tete-a-tete."
Sure I understand liability. But requiring consumer to buy another product to fully utilize they other product? Sorry but no. They have really cool hardware and nice UI but policy like that... no.
PeterStuer 21 hours ago [-]
Claude is quite good at producing an optimal robocopy command for your particular machines and target network use (you do not want 100% when copying dozens of TB over the LAN unless you want to annoy your family).
And when they work as advertised they are fantastic.
Huh, this is news to me but seems to check out from my Googling. I've really been enjoying the Ubiquiti ecosystem and was considering a UNAS to replace my Seagate NAS.
In the past 5 to 7 years they have abandoned a number of entire product lines and left the people running it hanging, with no way to purchase replacement parts, or keep a system running.
I believe I'll be more happierer as I integrate it to my life more and more.
Also a hint for anyone planning something similar: if you use Synologys SHR format (based on mdraid+btrfs) and your new destination supports or uses BTRFS too, you can just btrfs send your whole pool to the new NAS!
(Since its a single process it wont saturate any LAG though)
The fact that synology's btrfs is incompatible with a normal linux distribution and that the official instructions from synology for reading it is to use an older kernel together with ubuntu 16.04 lts convinced me to switch away from synology.
If the author is reading, it would be nice to clarify that the 10g network comment is irrelevant. I was under the impression at least 1 of the devices would have a 10g link, but at least 2 of the 3 didn't. It was a bit confusing, but it's also very late where I am.
Your suggestions is also very nice, you can just turn off your own PC and wait for it to finish.
I don't even think rclone can split across multiple interfaces like SMB does.
I just finished slurping all of my files off an old Synology and onto a TrueNAS server. Unlike the article here, my Syno is 2x1Gb in a LAG, so I was really only getting ~1Gbps at max.
One disappointment with TrueNAS is that it doesn't support SMB-MC unless you put each interface on a separate subnet (which is a strange limitation because Samba itself doesn't seem to suffer from this, since synology handles it fine)
With SMB you just plug in 2 NICs and you automatically get ~double the bandwidth.
Also, it kinda sounds like the UNAS might not be supporting ADS' for some reason? That alone could be a problem later on...
The whole reason of the post.
And the thing what linux fanbois with rsync/rclone here absolutely ignore.
I only know their higher end enterprise range specifically said ZFS but all the others doesn't seems to mention it. And I really wish there is a simple 4 bay ZFS NAS.
I don't know if this is better, but...an alternative is rclone, which uses rsync under the hood. It can also do parallel jobs and is restartable on any failed partial copy.
When I've had to do large Synology-to-Synology moves, that's what I've used. The multiple rsyncs in parallel really helps if you've got a lot of small files.
> rsync under the hood
Is it? I think one is written in C (rsync) and the other is in Go lang (rclone)
Something like :
will do nicely.It won't saturate 4 multichannel links, you'll most likely need rclone or rsync for that (or robocopy it appears), but it's much faster than rsync on initial copies.
Then for good measure, once the tar copy finishes, I run a single rsync verification between the two targets.
That being said, my latest storage migration was performed using either ChronoSync or Carbon Copy Cloner from my Mac mini server. They won't do anything rsync cannot already do, except present data and copy history in a graphical environment.
Be aware that tar doesn't preserve creation/birth times and keeping that original metadata is important to some people if the target file system can record it. Robocopy preserves/transfers creation times by default and recent rsync versions (3.2.3 2020-08-06) can do it with the extra option --crtimes.
Asynchronous IO, multithreaded for smaller files, serialized when they are big, block-level delta copying on subsequent runs. Pretty much as fast as it gets on Windows.
In other words, yes, it may be restartable, but if it takes more than twice as long, just recopy the file(s) if needed.
If you want to get the advantage of SMB multi stream copies (in rsync*), look for LACP or other options. Not sure if Synology does that though.
I'm honestly surprised that SMB was faster here. I've never had luck getting it to perform well, so this is a bit of a shock.
for example, Immich is a huge upgrade to Synology photos. I didn't know how much i was missing out.
https://docs.nersc.gov/services/bbcp/
My guess is the actual bottleneck would be directory traversal and metadata stuff - i.e. trying to keep the pipe full, not saturation effects.
Yeah I know on src/dest. I've done this sort of approach countless times in various technologies in the past 40 or so years, but that always puts the mechanism for checking things in onus of the human rather than having that one command that does it all right.
Anyway, not a slight on rsync in the slightest - it doesn't have this probably because no-one realistically actually needs it most of the time.
If they would allow me to send them email saying: "I hereby promise I wont record my neighbours having little tete-a-tete."
Sure I understand liability. But requiring consumer to buy another product to fully utilize they other product? Sorry but no. They have really cool hardware and nice UI but policy like that... no.