[RESOLVED] Copying large files across the network stops

[d3sys]

Hello, we tried to create a package for a business software suite, similar to the Adobe suite. The problem we encountered is that the installation is very large, so to manage any updates or bug fixes, we had to put everything on a NAS (via an SMB3 network share).
The current problem is that the installation fails because the folder is never completely copied to the client machine's location. We tried using Copytree and filecopy2 without success. Only `run_powershell copy-item` seems to work better, but it doesn't (sometimes Wapt completely skips the command line and copies nothing).

Wapt: 1.8.1
WaptServer: Debian 10
ConsoleWapt: Windows 10

[vcardon]

Hello, we tried to create a package for a business software suite, somewhat like the Adobe suite. The problem we encountered is that the installation is very large.

The largest package we know of is a National Instruments Labview with lots of modules, around 70GB, distributed using the classic method via a web download.

I advise against distributing your packages via SMB (see https://www.wapt.fr/fr/doc/wapt-common- ... takes.html), it's ugly.

There's no reason why you shouldn't be able to do it. Check your web server logs to see why it's blocking; you must have a technical limitation somewhere that you'll need to change.

Sincerely.

Vincent

[d3sys]

I advise against distributing your packets via SMB (see https://www.wapt.fr/fr/doc/wapt-common- ... takes.html), it's ugly.

Regards,

Vincent

Thanks for the link, but I'd already read it. The problem is that the documentation, while well-written and well-thought-out, is very brief and rather short when it comes to package creation. Most of the functions explained concern file manipulation, and what could be simpler than SMB for that?.

- Now this means we need to install a web server on the WAPT server. The question is: how do we code this cleanly to transfer an entire folder (approximately 80GB) after installation into our app's folder, and then launch the patches to be applied according to the user?
- The documentation (unless I missed it) didn't dwell much on file exchanges via WebServer, we don't know if it's a good idea to launch a Web server on the WAPT server.
- Does WAPT handle recovery in case of an HTTP/FTP connection failure?

[sfonteneau]

- Now this means that we need to install a web server in the WAPT server.

Wapt is already a web server that hosts Wapt packages. You don't need to install anything else.

The question is: how to properly code this to transfer an entire folder (approx. 80GB) after installation into our app's folder, and then launch the patches to be applied according to the user?
- The documentation (unless I missed it) didn't go into much detail about file exchanges via WebServer; we don't know if it's a good idea to run a web server on the WAPT server.

You just need to place the folder in your package

Code: Select all

c:\waptdev\tis-copyfile

Then in the package:

Code: Select all

def install():
    copytree2('sources','C:\\projet')

https://doc.tranquil.it/wapt/fr/doc/wap ... -directory

For code within the user session:

https://doc.tranquil.it/wapt/fr/doc/wap ... index.html

- Does WAPT handle recovery in case of an HTTP/FTP connection failure?

Yes, absolutely

[d3sys]

Wapt is already a web server that hosts Wapt packages. You don't need to install anything else.

You just need to place the folder in your package

Code: Select all

c:\waptdev\tis-copyfile

Then in the package:

That's precisely what we want to avoid. It took us almost 24 hours to send a 5GB package, and as for editing, well, don't even get me started. Imagine the urgent need to correct a programming error while it takes to download the file again before and after editing! That's why we opted for file sharing, which means the packages are much smaller (in kilobytes!) and there are no lengthy transfers just for a single code modification.
Unless we do it really wrong (by the way, we're programmers but not in Python) So we're still learning! Which I would truly understand... so I'm asking for clarification, our understanding is failing...
We have another problem with this method: during installation, subfolders don't seem to be taken into account. So I'd like to revert to a more practical (cleaner) method ) but which works effectively right away. So where is our mistake?

[vcardon]

https://www.wapt.fr/fr/doc/wapt-common- ... repository ?

+

https://winscp.net/eng/docs/task_synchronize

will resolve your issue.

Regards,

Vincent

[sfonteneau]

That's precisely what we want to avoid. It took us almost 24 hours to send a 5GB package
, and as for editing, well, don't even get me started. Imagine the urgent need to correct a programming error while it takes to download it again before and after editing? That's why we opted for file sharing, and as a result, the packages are much smaller (in kilobytes!) and there are no lengthy transfers just for a single code modification.

24 hours. It's really not Really That's not normal, is the server in the cloud or on the same local network?

We have another problem with this method: during installation, subfolders do not seem to be taken into account.

I just did a test with copytree2('sources','C:\\projet')

The subfolders are copied correctly

To put it simply, we use the "everything in the package" method precisely to avoid problems during installation. Copying errors, timeouts, network unavailability, etc...

There is also a security aspect; if everything is in the package, then everything is signed; if it is not in the package, then wapt cannot guarantee the integrity of the files since they are not in the package manifest.

For me, we need to check why your upload is so slow! Is it a bandwidth issue or something else?

[d3sys]

https://www.wapt.fr/fr/doc/wapt-common- ... repository ?

+

https://winscp.net/eng/docs/task_synchronize

will resolve your issue.

Regards,

Vincent

Thanks for the reminder. Not wanting to seem unpleasant or too stubborn, I returned with my team with better intentions... to start all over again with this valuable information...

Using WinSCP, upload your package to the /var/www/html/wapt/ directory on your Linux server.

Once the transfer is complete, you need to regenerate the Packages file in your repository:

wapt-scanpackages /var/www/wapt/

We upload/synchronize in www/html/wapt then we scan the packets contained in www/wapt/ Is this a mistake?

PM 24h This is really, really not normal

That's what we told ourselves... But apart from server overload (100 simultaneous users on average) I don't see what could cause this, hence the decentralization decision.

Is the server in the cloud or on the same local network?

The server is a Proxmox VM on a gigabit local network.

But the given solution, while it fixes the upload problem after build, does not completely solve the download problem for editing.
Let me explain:
- I understand that the files are always local and synchronized on the server.
- but
1- Each time the scripter is edited or launched, the last used setup is opened even if the package being edited is changed
2- The package version in control must therefore be updated manually (this must be specified in the documentation)
So, two easy sources of error... to err is human.
Don't you have a way to automate this process for large packages that need to be integrated/synchronized externally? (Nooo, we're not lazy!) just computer scientists in a room resembling an Egyptian tomb with a multitude of swords of Damocles just above our 3 cm of hair... (and time is against us...)

[florentR2]

If I understand correctly, what might help you would be to split your package into two parts.
On one side, the "heavy" files, and on the other, the scripts or configuration files that you need to modify frequently?
Then, make the package with the heavy files a dependency.

This way, you can quickly edit the package containing the scripts and configuration files, and it would also solve your problem with the subversion number incrementing automatically.

[vcardon]

And put the package with the large files as a dependency.

Excellent idea, I hadn't thought of that. Yes, that's what needs to be done.

Vincent