RSSAmplifier

Manish R Jain · Dec 6, 2022

Tranfer a Giant Directory across Network

0
Sign in to vote or save

This site does not allow itself to be embedded. You can still read it on the original site — the toolbar below keeps your place in the directory.

Originally Posted: Aug 24, 2022 Dealing with huge web3 data, I see myself doing this a lot: moving data directories across machines. In this case, we have a 250 GB directory with 10K files. The network connection between machines is ~5 MBps. But using these techniques, we can achieve a 6x faster transfer (~30 MBps). There’re two ways to do this: Use compressed tar (Newer and Older) Use rsync…

Read on manishrjain.com

Comments

Nothing yet. Say the first thing.

    Sign in to join the conversation.