Tjyylli

Why did Kant, Hegel, and Adorno leave some words and phrases in the Greek alphabet?

Greatest common substring

How to be diplomatic in refusing to write code that breaches the privacy of our users

Teaching indefinite integrals that require special-casing

Student evaluations of teaching assistants

Everything Bob says is false. How does he get people to trust him?

How can I get through very long and very dry, but also very useful technical documents when learning a new tool?

How does it work when somebody invests in my business?

Can somebody explain Brexit in a few child-proof sentences?

Using parameter substitution on a Bash array

How do I keep an essay about "feeling flat" from feeling flat?

Is there a good way to store credentials outside of a password manager?

How can I replace every global instance of "x[2]" with "x_2"

Curses work by shouting - How to avoid collateral damage?

Why does John Bercow say “unlock” after reading out the results of a vote?

Personal Teleportation as a Weapon

What are the ramifications of creating a homebrew world without an Astral Plane?

Go Pregnant or Go Home

Was Spock the First Vulcan in Starfleet?

What would be the benefits of having both a state and local currencies?

Irreducibility of a simple polynomial

At which point does a character regain all their Hit Dice?

What is the opposite of 'gravitas'?

Can criminal fraud exist without damages?

Automatic one-way syncing of a Windows file share to an S3 bucket

Rsync, two-way sync, and syncing file-deletion operationsCross-platform file syncing over LANOne-way syncing/backup with UnisonReplacing Dropbox with: Amazon S3 + SSL + GPG/TrueCrypt + Mounting on OSX ??Good one-way sync program for Windows?Two-way syncing folders between PCs and smb shareIs there a way to automate XML playlist creation for files in an S3 bucket?one way bookmark syncing/syncing issues ChromeMultiple file server syncingmultiple s3 users in one bucket or per-user buckers

0

Our clients have Windows file shares where they store lots of different documents.

We want to ingest files matching wildcard patterns (e.g. *.pdf, *.xlsx, etc.) from those file shares into S3 buckets for purposes of building a data lake.

We're considering a bunch of alternatives, but we feel like some people could have better ideas how to do this:

1. 
   

   Run agent software on the clients' Windows file servers.

   
   
   

   1.a. Write a PowerShell script that invokes rclone to copy new / modified / deleted files to S3 and call a reporting HTTP API to let us know whether it succeeded or failed, and schedule that script to run periodically using the task scheduler.

   
   
   

   1.b. Do the same thing, but using aws s3 sync from aws-cli.

   
   
   

   1.c. Homeroll our own uploader based on Facebook Watchman or similar file watching solution.

   
   



2. Setup an SFTP or WebDAV server on the client file server and run rclone from an EC2 instance we manage.

We want to know if there's no other simpler technology we could use. Could Amazon FSx help somehow? Like some kind of automatic replication mechanism for mirroring our client file shares on Amazon FSx Windows file systems?

Or maybe, is there some other kind of "roll your own Dropbox using an Amazon S3 backend" solution?

windows sync mirroring amazon-s3

share

asked 7 mins ago

Guilherme VieiraGuilherme Vieira

1185

add a comment | 

0

Our clients have Windows file shares where they store lots of different documents.

We want to ingest files matching wildcard patterns (e.g. *.pdf, *.xlsx, etc.) from those file shares into S3 buckets for purposes of building a data lake.

We're considering a bunch of alternatives, but we feel like some people could have better ideas how to do this:

1. 
   

   Run agent software on the clients' Windows file servers.

   
   
   

   1.a. Write a PowerShell script that invokes rclone to copy new / modified / deleted files to S3 and call a reporting HTTP API to let us know whether it succeeded or failed, and schedule that script to run periodically using the task scheduler.

   
   
   

   1.b. Do the same thing, but using aws s3 sync from aws-cli.

   
   
   

   1.c. Homeroll our own uploader based on Facebook Watchman or similar file watching solution.

   
   



2. Setup an SFTP or WebDAV server on the client file server and run rclone from an EC2 instance we manage.

We want to know if there's no other simpler technology we could use. Could Amazon FSx help somehow? Like some kind of automatic replication mechanism for mirroring our client file shares on Amazon FSx Windows file systems?

Or maybe, is there some other kind of "roll your own Dropbox using an Amazon S3 backend" solution?

windows sync mirroring amazon-s3

share

asked 7 mins ago

Guilherme VieiraGuilherme Vieira

1185

add a comment | 

Our clients have Windows file shares where they store lots of different documents.

We want to ingest files matching wildcard patterns (e.g. *.pdf, *.xlsx, etc.) from those file shares into S3 buckets for purposes of building a data lake.

We're considering a bunch of alternatives, but we feel like some people could have better ideas how to do this:

1. 
   

   Run agent software on the clients' Windows file servers.

   
   
   

   1.a. Write a PowerShell script that invokes rclone to copy new / modified / deleted files to S3 and call a reporting HTTP API to let us know whether it succeeded or failed, and schedule that script to run periodically using the task scheduler.

   
   
   

   1.b. Do the same thing, but using aws s3 sync from aws-cli.

   
   
   

   1.c. Homeroll our own uploader based on Facebook Watchman or similar file watching solution.

   
   



2. Setup an SFTP or WebDAV server on the client file server and run rclone from an EC2 instance we manage.

We want to know if there's no other simpler technology we could use. Could Amazon FSx help somehow? Like some kind of automatic replication mechanism for mirroring our client file shares on Amazon FSx Windows file systems?

Or maybe, is there some other kind of "roll your own Dropbox using an Amazon S3 backend" solution?

windows sync mirroring amazon-s3

share

asked 7 mins ago

Guilherme VieiraGuilherme Vieira

1185

share

asked 7 mins ago

Guilherme VieiraGuilherme Vieira

1185

share

asked 7 mins ago

Guilherme VieiraGuilherme Vieira

1185

asked 7 mins ago

Guilherme VieiraGuilherme Vieira

1185

asked 7 mins ago

Guilherme VieiraGuilherme Vieira

1185