Author Topic: IDEA: Any program required to move files to a NAS. (Read 30,271 times)

mouser · « **Reply #25 on:** June 24, 2018, 06:45 PM »

Quote

I'd like to copy/move the file as safe as possible, so using Operating System native tools has my preference.

Agreed.

4wd · « **Reply #26 on:** June 25, 2018, 03:07 AM »

RoboCopy was designed to be safe, can copy anything related to a file, (dates, ADS, security, etc), is multi-threaded, can skip corrupted files, can move or copy, and is native to the OS.

An option might be to use the GUI to set up the parameters and call the RoboCopy command and parse its output back into the GUI output window?

wraith808 · « **Reply #27 on:** June 25, 2018, 03:29 AM »

^ +1. We really need a way to endorse posts so we don't have to just post that we agree...

4wd · « **Reply #28 on:** June 25, 2018, 06:11 AM »

^ +1. I agree

KodeZwerg · « **Reply #29 on:** June 25, 2018, 07:25 AM »

You mean integrate calls to commandline tool "robocopy.exe"?
Hmm.... i could integrate an option to call external Tools with a checkbox to show console.
Option to add commandline parameter that sticks to external Tool would be neccessary aswell i guess.
If external app support %ErrorLevel% as Result i guess i could intgrate a check.

Okay, will respect it and add such option! (thanks god that i already switched to a single configuration dialog

)

edit
After re-reading this thread, i must admit the important information i havent understood at first time reading, now its clear! Sorry for that!
(I'd thought "I'd like to copy/move the file as safe as possible, so using Operating System native tools has my preference." = means use Api calls)

mouser · « **Reply #30 on:** June 25, 2018, 09:36 AM »

Quote

I'd thought "I'd like to copy/move the file as safe as possible, so using Operating System native tools has my preference." = means use Api calls

That's what I meant -- to use the Windows API functions for CopyFile or CopyFileEx or the delphi file copy functions.

Craig381 · « **Reply #31 on:** June 25, 2018, 09:49 PM »

Thanks for joining in on my app request all.
KodeZwerg is doing a lot of work in order to save me hours of sorting through files and me having to move them 1 at a time to multiple drives.

To Ath, my app request requirements are clearly laid out by in my initial request and yes, I have considered my NAS structure over and over again to the nth degree involving hundreds of hours of testing and after ruining several hard drives, RAID is out of the question and to be quite honest, RAID is not a feasible solution in todays large size hard drive environment as re-writes take too long, jeopardize server up time and RAID itself wears out drives with all the handshaking and checking and re-writes.

I am presently using NAS4FREE and JBOD with UFS why JBOD? because I have 2 offsite copies of all files.
Why NAS4FREE, it's free, easy to use, and has been working reliably for several years now as my NAS grows in size.

There is an old data storage rule which has never changed. The rule of 3, 3 copies of all data, of which 1 must be at another location.
RAID simply does not accomplish this and I have followed this rule... starting right after this story below happened to me.

PS: RAID is a sensitive issue with me as I paid $1000.00 USD for 2 IBM state of the art 7200rpm Deathstar 13GB HDD's several years back. They worked in RAID config for about 2 days before overheating and going bad. Yes I was kickass fast in a MB world for a couple days! I sent them both back 3 times (all new ones failing again and again) taking 8 weeks transit time each time and on the 3rd time they came back they told me the warranty had finally expired and that they do not recommend leaving the HDD on for more than 8 hours at a time..lol and that there would be no more exchanges. I joined the class action law suit and to this day have not received one cent.

Shades · « **Reply #32 on:** June 26, 2018, 04:13 AM »

Then you might be interested in SnapRAID. It has tickled my interest, if I'm honest. It promises practically all the speed advantages of a software RAID setup, but none of the disadvantages.

[rant]
Not a fan of RAID in general though. Mainly because I have inherited a software RAID setup (Linux), that often took so much time to rebuild that it would have been easier/faster to start from scratch from a backup. A few times even 48 hours or thereabout. And during those rebuilds you can't use that setup, so people started to work around it altogether. But the boss drunk from the cool-aid: "but data is much safer, because you can rebuild". So I must keep it alive.

In my view, the only thing RAID is good for: speed
But when (not if) problems arise, it will take much more effort and money to access data from drives that were part of a RAID setup....and that is if you are lucky. Most of the time content is not salvageable. Granted, that is a bigger problem with hardware RAID solutions than with software RAID solutions. Hardware RAID is practically always faster than a software RAID setup, so such solutions remain preferred by most. Yet those persons/companies forget that they must have a redundant identical hardware setup ready if they take their data serious. Or at least double the amount of (identical!) RAID cards, because if you don't, you easily reduce your already slim chance of getting any data back by 80%.

In short, a heavy price tag to slap on data. Companies that absolutely need all the speed for their systems to be able to do their core business without losing money, should consider RAID. All others can wait a second or two more for their data. Besides, nowadays more and more node-based software is appearing that is very fast from itself and truly excels when the network they are connected on is super fast. No hardware RAID can beat those speeds and these systems only need standard machines with standard OS's that are relatively easy to repair/exchange, without downtime. NoSQL databases like Apache Cassandra are a good example of this.

RAID is a yesteryear mindset. But, if you must, I do think that the concept behind SnapRAID is worthy of consideration.
[/rant]

Ath · « **Reply #33 on:** June 26, 2018, 10:03 AM »

Quote from: Craig381 on June 25, 2018, 09:49 PM

yes, I have considered my NAS structure over and over again

Well, that was the type of answer I was looking for: You have a very well motivated reason to continue in this way. Thnx, but it wasn't obvious from your OP.

Craig381 · « **Reply #34 on:** June 26, 2018, 09:14 PM »

Instead of "hard coding" the destination folders why not..

Use cmd net view \\192.168.1.203 to obtain Samba shares on the NAS
pipe all the shares into a list
dir each item on the list one at a time to obtain the folder names within each share
pipe them into a list and use that list to make the "rules" to move the files to the proper folders.
This way the shares and folders on the NAS can "change at will" and it will not affect the program moving the files.

anyone ever try something like this?

Craig381 · « **Reply #35 on:** August 16, 2018, 02:59 PM »

I was initially so excited for the initial response.

This apparently has been forgotten.

Guess I will have to write my own program.

sigh

KodeZwerg · « **Reply #36 on:** August 16, 2018, 07:24 PM »

Hello Craig381,

i feel sorry to silent dropped this discussion.
Initial i really felt to help, and it was already finished, at least 95% with maybe 5% bugs left.
But that "do this/dont that" and "use this/not that" has killed my fun in developing it, its already dumped after all those "delete is a no go" texts.

That RoboCopy thing gave me the rest to forget about this project because i would not have 100% control, or i am not skilled enough.

If i am again motivated enough (not now, we had a bad accident in family), i still can do the job by ignore everything and just do my thing.

4wd · « **Reply #37 on:** August 17, 2018, 01:05 AM »

@Craig381: Does the NAS require log on credentials or are they already stored/saved in Windows?

ie. Can you just access the share through Explorer without entering user/password?

Also, define "special characters", is this anything other than what Windows defines as forbidden filename characters, (ie. <>: "/\ |?*) ?

ADDENDUM: Also, you could ask skwire to enhance his File Punter program to accept UNC destinations, it should be able to do what you want after you create the RegEx rules.

skwire · « **Reply #38 on:** August 17, 2018, 10:53 PM »

Quote from: 4wd on August 17, 2018, 01:05 AM

ADDENDUM: Also, you could ask skwire to enhance his File Punter program to accept UNC destinations, it should be able to do what you want after you create the RegEx rules.

Oddly enough, I was going to ask @Craig381 if he would be open to using my ZeroZipper program on some of his file structure for me to test with to see if I could make FilePunter work for his needs. @Craig381, if your filenames are sensitive, we understand. However, if you are you open to this, you can use my ZeroZipper program to make a zero-byte mirror zip of your source files for me to test with.

4wd · « **Reply #39 on:** August 17, 2018, 11:11 PM »

I think it would, creating RegEx rules to cater for them is easy.

Trying to programmatically create them from the directory names is a little more of a challenge.

Kind of getting there in PowerShell but i don't have a lot of time available at the moment.

Craig381 · « **Reply #40 on:** August 18, 2018, 04:17 AM »

\\192.168.1.203 is enough in windows explorer to view NAS drives/shares.
Login is already accomplished.

This "timegap" is exactly what I needed to get off my arse and start coding.
In the past month I have managed to get a semi-working vbscript wrapped in a hta gui which is not quite there yet...

It presently takes the global variable NASPath = 192.168.1.203 and learns the entire NAS structure thru a seperate running process using a simple net view cmd saved to a txt file which is parsed and saved into 1 dimension of a 3 dimensional array. I then take the number of subfolders and names of all said subfolders on each drive and save them into another dimension of the array. Then came the tough part which took the past 2 weeks, figuring out all the possible regex patterns that could exist, extracting and writing them from actual names stored in the array and then writing the "dynamic" patterns back into the array so that they correspond with the actual subfolder names on each NAS drive.
a lot of crap like this.. If RE.Pattern= bla bla Then N = "^(" & Left(d,2) & ")[" & Right(Left(d,3),1) & "-" & Right(Left(d,7),1) & "].*"
Finally I saved the array into a config.ini file to speed up the second running of the program.
This was tougher than I thought, but I am at 689 lines of code and Im not gonna quit till this damn thing works.... (unless someone beats me to it then I will pause to admire their work before finishing this monstrosity.)

Craig

Craig381 · « **Reply #41 on:** September 13, 2018, 04:00 PM »

After reading back thru the posts I am amused that KodeZwerg stopped because of comments NOT MADE BY ME.

KodeZwerg quote: "But that "do this/dont that" and "use this/not that" has killed my fun in developing it, its already dumped after all those "delete is a no go" texts."
These comments were ALL OTHER PEOPLE and NOT MADE BY ME, the originator.

What kind of person punishes person A because of what person B & C & D say or do?

Bottom line, shame on you KodeZwerg for bailing (unless you were unable to technically accomplish it. then I am sorry.)

ps: I still have a coder donation for anyone who is able to accomplish this.

It is actually now a CODER CHALLENGE as my program is finished... 700 lines of code and it works!

oh, my program follows no set file structure as stated in the first post, forget all that crap, it learns the shares and subfolders on the network and moves all files residing in the program folder to where they "should" belong on the network according to the first 3 characters in each file name. There is a "Re-Learn Network" button (For changes to the NAS), a "Replace if Larger" button and a "Use Previous Config" button which also moves duplicate files to the G:\Duplicates folder. (The Larger files option also moves smaller files here.)

wraith808 · « **Reply #42 on:** September 13, 2018, 05:42 PM »

Quote from: Craig381 on September 13, 2018, 04:00 PM

Bottom line, shame on you KodeZwerg for bailing (unless you were unable to technically accomplish it. then I am sorry.)

Shame on him? He bailed because he thought that was a general consensus. Reach out to him directly to say, "Hey, I don't care. Can you complete this?"

Craig381 · « **Reply #43 on:** September 14, 2018, 12:23 AM »

Listen here Wraith, how the hell do you know what someone else is thinking?
Don't tell me what to do and how about you mind your own damn business.

This has nothing to do with you.

I would like to see new rules on this site that only allow the originator to dictate what they want or require because all the side comments ruined this with their OWN personal demands/suggestions which has NOTHING to do with me.

So get that into ya..

mouser · « **Reply #44 on:** September 14, 2018, 12:56 AM »

The coders who hang out and participate here, including KodeZwerg, do so because it brings them some satisfaction to interact with users of their software. If those efforts become unsatisfying it is not unreasonable for them to stop working on it. This early on a project it can be a very wise thing to stop investing time in something that isn't making you happy. There are just too many projects calling out for our time and attention, sometimes we have to choose carefully what projects get our attention and which have to be let go.

wraith808 · « **Reply #45 on:** September 14, 2018, 01:25 AM »

Quote from: Craig381 on September 14, 2018, 12:23 AM

Listen here Wraith, how the hell do you know what someone else is thinking?
Don't tell me what to do and how about you mind your own damn business.

This has nothing to do with you.

I would like to see new rules on this site that only allow the originator to dictate what they want or require because all the side comments ruined this with their OWN personal demands/suggestions which has NOTHING to do with me.

So get that into ya..

I never said anything about you, I talked about what he had said, and a more reasoned approach that you might take to get his attention instead of attempting to shame him into continuing or shame him for not continuing. I was actually considering looking at this if he didn't want to continue because I could see that he dropped it because of crosstalk, not as a punishment to you, but in his own words:

Quote from: KodeZwerg on August 16, 2018, 07:24 PM

i feel sorry to silent dropped this discussion.
Initial i really felt to help, and it was already finished, at least 95% with maybe 5% bugs left.
But that "do this/dont that" and "use this/not that" has killed my fun in developing it, its already dumped after all those "delete is a no go" texts.

Try to motivate him, not shame him was my only suggestion. Whatever you take from that is your problem. And no, I'm in no way thinking of picking this up at this point. Onward!

Craig381 · « **Reply #46 on:** September 16, 2018, 04:40 PM »

Wraith808, you are full of crap. Read back thru the posts, Everything was going great until you and Mouser (who should know better) and 4wd and Ath just had to chime in and just couldn't stay out of this initial post and practically hijacked it with all your own input.

I felt like reaching thru the screen and telling you all to stfu. This was my requirement/idea NOT yours for gawd sakes.

All of you should be ashamed of yourselves for butting in and ruining this.

I am the one who did not get anything, YES I am the one who is punished here, thats the bottom line.

Ring the shame bell.

wraith808 · « **Reply #47 on:** September 16, 2018, 05:11 PM »

Quote from: Craig381 on September 16, 2018, 04:40 PM

Wraith808, you are full of crap. Read back thru the posts, Everything was going great until you and Mouser (who should know better) and 4wd and Ath just had to chime in and just couldn't stay out of this initial post and practically hijacked it with all your own input.

I felt like reaching thru the screen and telling you all to stfu. This was my requirement/idea NOT yours for gawd sakes.

All of you should be ashamed of yourselves for butting in and ruining this.

I am the one who did not get anything, YES I am the one who is punished here, thats the bottom line.

Ring the shame bell.

I didn't jump in, other than to agree that robocopy was a good tool. It's apparently you that aren't reading well. You could indeed reach out to KodeZwerg behind the scenes to see if he'd be willing to continue. Offer him encouragement and perhaps even compensation. But instead, you'd rather play the victim, and try to shame others- even the one that was trying to help you. Entitled much?

mouser · « **Reply #48 on:** September 16, 2018, 05:32 PM »

The nature of this site is such that we all jump into each other's threads, make suggestions, offer to help, get bored and leave threads, offer unsolicited advice, etc. Just think of it as a large family with an assortment of all types of people. It's just how it is.. This is our hobby not our job.

KynloStephen66515 · « **Reply #49 on:** September 16, 2018, 05:39 PM »

This is a public post, on a public forum. People will obviously bring their own thoughts and ideas to the topic as that's how this forum works.

If this thread continues to dissolve into petty arguments and name calling; it will be locked.

Please consider this as the final "off-topic" message in this thread, and get back to the original topic, rather than letting madness descend.

~Stephen.