File Server Synchronization with Office 365, SharePoint & OneDrive

 

Documents on traditional file servers or NAS like department shares or user home drives can be centrally synchronized with Microsoft Office 365, SharePoint, OneDrive for Business, Office 365 Groups, or Microsoft Teams using the Layer2 Cloud Connector with very flexible options. The connector can close many gaps with SharePoint and Office 365 out-of-the-box document synchronization, e.g. regarding the number of files allowed, file naming conventions, file types allowed, document set selection, metadata assignment, central management, logging, alerting, and more. Please take a look here for more supported systems and applications like Google Drive, Dropbox, or Box.

 

Layer2-File-Sync-600.png

 

Fig.: The Layer2 Cloud Connector can solve many issues that still exist with on-premises to cloud document migration, backup, and synchronization.

 

When you are moving to the Microsoft Office 365 or SharePoint Online cloud, there are a lot of reasons to keep your local file servers in sync:

 

  • Be compatible with non cloud-based applications, legacy systems or business processes (e.g. workflows) that require local access.
  • Work offline with no restrictions, or in the corporate office with increased performance and without internet bandwidth limitations - but also online while traveling.
  • Archive files locally for compliance reasons. Synchronization can provide a local backup of the cloud-based files, you can archive files using common backup tools.

 

In some cases it could also make sense to backup your local files to the Microsoft cloud, e.g. to benefit from SharePoint document versioning for archive.

 

The Layer2 Cloud Connector has the following advantages compared to Microsoft out-of-the-box offerings:

 

  • Works with any Microsoft cloud service or plan: Microsoft Office 365, SharePoint on-prem and Online, OneDrive for Business, Office 365 Groups, Microsoft Teams, and others. 3rd party services like Google Drive, DropBox, Box.NET, and others can be integrated via local shares.
  • Centrally managed server-to-server sync tool - Windows Service, no Office-like end-user client.
  • Very flexible sync options like support of local directories, NAS file shares, target library subdirectories, additional no-insert / no-update / no-delete options. Uni- or bi-directional sync. Sync files or just metadata.
  • There is no hard limit for the amount of synced files (e.g. 5.000 / 20.000 list view threshold with OOB). But take care about bandwidth limitations, especially for first sync. Only changes are synchronized later on within seconds or very few minutes (depending on  amount of changes). Please edit your SharePoint views to still be able to navigate your files: Remove all sorting, filtering, grouping. Also set to flat view in the folder section. You can also make use of SharePoint search or managed metadata to find your files.
  • You can keep your existing file server directory structure: No extra OneDrive sync folder required.
  • SharePoint file name issues solved (requires V5.2 or higher). Special characters are replaced by default.
  • SharePoint file type issues solved (requires V5.2 or higher). Forbidden types are zipped automatically.
  • SharePoint empty file issues solved (requires 5.2 or higher).
  • Flexible column mapping, e.g. for mapping of file meta data to custom SharePoint list columns.
  • SharePoint view support during sync to sync several sources into one library. Include or exclude SharePoint files or folders to / from sync.
  • SQL support for file system queries (requires 6.2 or higher). You can e.g. include or exclude specific date ranges (e.g. to sync just new files), specific file types, files with specific names or paths, specific file sizes (e.g. sync just files below 1 GB). See User's Guide for the detailed SQL options and syntax.
  • Dynamic Columns to add custom metadata or other content on-the-fly via C#.
  • Several different options to solve replication conflicts. General error management.
  • Flexible authentication options, e.g. with specific Windows accounts or Windows integrated, Office 365, Active Directory Federated (ADFS), or custom (using the Authentication Construction Kit).
  • PowerShell scripting can be applied to automate creation and execution of connections, e.g. for migration of a large user base to OneDrive for Business, or to backup all libraries of a SharePoint site or site collection.

  • Advanced logging options via NLOG (e.g.to files, SQL, Windows) including notifications / alerting. Several different log levels.
  • File synchronization is just one option. It can sync data and files also SharePoint to SharePoint or with almost any other system or application, e.g. databases.
  • Cost effective licensing per installation: No user client access licenses (CALs) required. No volume-related costs. Multiple connections to different teneants, servers, sites, or libraries.

File Share to Office 365 Synchronization Specific Settings

 

​The Layer2 Cloud Connector must be configured as follows to access a local or network file share.

 

Sample configuration for file share synchronization

 

Fig.: Sample connection configuration to connect to a local directory C:\Test2 for synchronization.

 

Please note the following specific settings.

 

  • Select the Layer2 Data Provider for File System to connect. The provider is part of the Layer2 Cloud Connector distribution and should be already installed, depending on installation options selected. You can also install separately if required.
  • Note that the driver architecture must fit to the Cloud Connector version installed (32- or 64-bit). 
  • You can use a connection string like this to connect:
    Directory=myDirectory; UID=myUser; PWD=myPassword
    myDirectory can be D:\myDir, \\myServer\myshare, or \\127.0.0.1\c$\myShare (with any valid IP).
    Don't use H:\ drive notation in case you receive a "bad net name" error (using a scheduled sync).
    myUser usually is a domain account like myDomain\myUser. 
  • SQL-like data queries are supported, e.g. only files within specific folders, with specific names, types, size or age.
  • You can map your source data fields (metadata like created, modified) to specific external source fields in the Layer2 Cloud Connector. Please take care about data types (simple type conversions are supported).
  • No primary key required to enter manually.
  • You can use the connection for uni- or bi-directional synchronization. Several replication conflict solution strategies are offered.
  • No installation or changes are required at the data source or data destination.
  • Data synchronization can be started manually in the Connection Manager, per command line or scheduled in background by the Layer2 Cloud Connector Windows Service. Only data changes are processed (no delete / bulk import). Metadata added in SharePoint are kept during updates.
  • No programming required for setup a connection and sync.
  • No need to open your local network for access from outside.
  • Currently supported syncs: File Share to File Share, File Share to SharePoint / Office 365 / OneDrive for Business, SharePoint to SharePoint, e.g. local to cloud).

Known Issues & Workarounds

 

There are still some known issues with Office 365 document synchronization. You can workaround by following the best-practice advises below:

 

  • The first sync (e.g. migration to the cloud) must be managed by the tool itself to ensure integrity (manual upload / download or other ways do not help).
  • In case of cloud migration, best to start with an empty SharePoint library, as existing documents in the target are removed or overwritten (uploaded again). Deleting documents in SharePoint via API can be time-consuming for any reason. Remove manually to save time.
  • You can include or exclude documents and folders to / from synchronization very flexible, Make use of SQL-like queries on the file server side. Make use of views on the SharePoint side (view=myView parameter in connection string, see User's Documentation).
  • Take care about bandwidth limitations and Office 365 throttling, especially for the first sync. Estimations are displayed if using the toolbox for first sync. Estimations can change during the sync process, stay patient. You can stop and restart the sync whenever you want.
  • Only changed files are synchronized later on within seconds or a few minutes (depending on amount of changes, items, connections, and configuration).
  • Take care about warnings and errors during first sync. In most cases it is about naming conventions and other issues that can't be solved by the tool itself. You can make use of a Microsoft tool to find and fix issues before first sync. See User's Documentation, how the connector manages some typical issues.
  • Expect some seconds communication overhead per single document (depending on configuration), additional to the file upload / download itself.
  • While there is no hard limit for the amount of files to sync, keep the number of documents per library (connection) as low as possible. To keep 100.000 documents in sync should not be problem, a powerful configuration can do much more. But consider the SharePoint / OneDrive limitations, and your Office 365 plan.
  • Note about the SharePoint list view threshold (above 5.000 / 20.000 documents per library). This is not an issue for the sync - but could be an issue for some SharePoint features and apps. Users will still be able to page through a library, use search to find specific files. There can be issues with using specific views, filters, grouping etc. See Microosft documentation for more details. 
  • Above the list view threshold use a specific view for the data sync. Address the view using the view-parameter in connection string. Remove all sorting, filtering, grouping.
  • Sync to a specific new root folder in a SharePoint library is not allowed above the list view threshold (as unfortunately folders are implemented as filters in SharePoint).
  • Large files are streamed to SharePoint to keep memory allocation as low as possible. Anyway, parts of the file and also metadata must be kept in memory. Take care about memory utilization in case of any issues.
  • Take care of the update interval. Keep it high (e.g. 1h) if you don't need to be up-to-the-minute (e.g. for backup). Keep it low in case changes are made on both sides to avoid possible conflics. The update interval should be higher as the time a typical update needs to complete. 15-20 minutes is a good starting point to adjust later on.

Step-by-Step Intros File Server to Office 365 Document Synchronization

 

​Just some ideas about systems to connect and sync:

 

  • Sync local or network shares with SharePoint or Office 365.
  • Sync local or network shares with OneDrive for Business.
  • Make use of the Layer2 Cloud Conenctor for document migration (one-way sync). Note there are no volume-related costs. You can use PowerShell to script the migration tasks.
  • Make use of the Layer2 Cloud Connector to backup cloud data to file shares in case you need it locally for compliance reasons.
  • Use it for backup of local files to a OneDrive for Business library. Note that you have versioning available in the cloud - great to restore older versions.
  • Sync file shares with other file shares, locally, on file servers or NAS.
  • Sync any 3rd party cloud storage (e.g. BOX) to local file shares as intermediate store (with 3rd party tools) and than sync to SharePoint, Office 365 or OneDrive.
  • Sync document metadata only to SharePoint lists (not the files itself) and have search, change notifications, workflows. Link back via FTB, WebDav etc.
  • Update document libraries in SharePoint, Office 365 and OneDrive 4B with metadata from SQL/ERP/CRM. You can use Dynamic Columns and C# to retrieve, transform, and apply custom metadata.

 

Do you have more ideas? Just let us know. Can't find what you are looking for? Please contact [email protected] directly for more information.

 

 

Next Steps For Evaluation

 

​​​​​You can register to download and evaluate the Layer2 Cloud Connector here.

FIND RELATED SOLUTIONS HERE.

Learn more about similar solutions for data migration, backup, integration, and synchronization by vendor and topic.

Ready to go next steps?

Register for free download.

Keep your Sharepoint in sync. Download and try today.

Questions? Contact us.

We are here to help. Contact us and our consulting will be happy to answer your questions.