SharePoint Rsync List: How to Sync Files Between SharePoint and Linux

Automating SharePoint Rsync List Tasks: Practical Scripts & ExamplesAutomating synchronization between SharePoint and non-Windows environments can save time, reduce errors, and make collaboration seamless across platforms. This article shows practical approaches for building an automated “SharePoint rsync list” workflow: collecting lists of files from SharePoint, mirroring those files to a Linux or macOS host using rsync-like transfer behavior, and automating the end-to-end process with scripts and scheduling. It covers the concepts, authentication options, sample scripts (PowerShell, Python, and Bash wrappers), error handling, performance tips, and security considerations.

---

What “SharePoint Rsync List” Means in Practice

SharePoint stores files inside document libraries accessible via HTTP(S) endpoints and APIs rather than as a native POSIX filesystem. “Rsync list” in this context refers to generating a list of files (with metadata and paths) from SharePoint and using an rsync-style synchronization approach to copy, update, or delete files on a remote Unix-like host so the target mirrors SharePoint content.

The main steps are:

• Authenticate to SharePoint (OAuth, App Registration, or credentials).
• Enumerate files in one or more document libraries.
• Download new/changed files to a local staging area.
• Use rsync (or rsync-like behavior) to synchronize files to the final target.
• Optionally upload changes back to SharePoint or reconcile deletions.

---

Architecture options

There are several common architectures for automating this workflow:

1. Direct API-based sync

   • Use Microsoft Graph or SharePoint REST API to enumerate and download/upload files.
   • Pros: full control, supports metadata, permissions, and large files (with chunked upload).
   • Cons: requires handling API rate limits, authentication.

2. WebDAV / WebClient

   • Mount SharePoint as a network drive (WebDAV) and then use native rsync.
   • Pros: simple once mounted.
   • Cons: WebDAV on SharePoint often has quirks, locking, and poor performance for large libraries.

3. Hybrid (staging + rsync)

   • Use API or tools to pull files into a local staging directory, then run rsync to any Unix host or NAS.
   • Pros: robust, allows batching, compression, and delta transfers on the final hop.
   • Cons: requires additional storage and an intermediate step.

---

Authentication options

• OAuth 2.0 with Azure AD App Registration (recommended for production)

  • Use client credentials (app-only) for server-to-server automation.
  • Grants appropriate Graph/SharePoint permissions (Sites.Read.All, Sites.ReadWrite.All).

• Username/password (legacy)

  • Feasible for small scripts; less secure and often blocked by modern tenants.

• Device code / Interactive

  • Useful for one-off or admin-run scripts; not suitable for headless automation.

• NTLM / Kerberos (on-premises SharePoint)

  • For intranet environments where the server supports Windows authentication.

---

Practical examples

Below are practical, runnable examples showing how to:

• enumerate a SharePoint document library,
• build a file list suitable for rsync,
• download changed files,
• use rsync to synchronize to a remote Linux host.

All examples assume you have permission to access the SharePoint site and the document libraries.

---

Example 1 — PowerShell: Enumerate files + download via PnP.PowerShell

This PowerShell approach uses the PnP.PowerShell module which wraps SharePoint REST calls and handles authentication. It’s convenient on Windows or PowerShell Core on Linux/macOS.

Prerequisites:

• Install-Module PnP.PowerShell
• Register an Azure AD app if running non-interactively (or use interactive Connect-PnPOnline)

# Connect interactively (for testing): Connect-PnPOnline -Url "https://contoso.sharepoint.com/sites/Team" -Interactive # Or use App-Only with certificate or client secret: # Connect-PnPOnline -Url "https://contoso.sharepoint.com/sites/Team" -ClientId $clientId -Tenant $tenantId -ClientSecret $secret $library = "Documents" $localStaging = "/tmp/sharepoint-staging" New-Item -ItemType Directory -Path $localStaging -Force | Out-Null # Recursively get files $files = Get-PnPListItem -List $library -PageSize 500 -Fields "FileRef","Modified","FileLeafRef" -ScriptBlock {     param($items)     $items } foreach ($item in $files) {     $fileRef = $item["FileRef"]     $fileName = $item["FileLeafRef"]     $remotePath = Join-Path $localStaging ($fileRef.TrimStart("/sites/Team/"))     $dir = Split-Path $remotePath -Parent     New-Item -ItemType Directory -Path $dir -Force | Out-Null     # Download file     Get-PnPFile -Url $fileRef -Path $dir -FileName $fileName -AsFile -Force } 

After files are in the staging folder, use rsync to push to a Linux host:

rsync -avz --delete /tmp/sharepoint-staging/ user@linuxhost:/var/www/sharepoint-mirror/ 

Notes:

• The script preserves folder structure by using FileRef.
• Use –delete to mirror deletions; be careful — deletions on SharePoint will remove remote files.

---

Example 2 — Python + Microsoft Graph: build a list and download changed files

Python example using Microsoft Graph API and requests. For production, use MSAL for authentication.

Prerequisites:

• pip install msal requests

import os, requests, msal, json, hashlib from urllib.parse import quote TENANT_ID = "your_tenant_id" CLIENT_ID = "your_client_id" CLIENT_SECRET = "your_client_secret" SITE_ID = "your_site_id"  # obtain via Graph explorer or API DRIVE_ID = "your_drive_id"  # document library drive id, or use /sites/{site-id}/drives STAGING = "/tmp/sp-staging" os.makedirs(STAGING, exist_ok=True) # Acquire token (client credentials) authority = f"https://login.microsoftonline.com/{TENANT_ID}" app = msal.ConfidentialClientApplication(CLIENT_ID, authority=authority, client_credential=CLIENT_SECRET) token = app.acquire_token_for_client(scopes=["https://graph.microsoft.com/.default"]) access_token = token['access_token'] headers = {"Authorization": f"Bearer {access_token}"} def list_children(drive_id, item_id=None):     if item_id:         url = f"https://graph.microsoft.com/v1.0/drives/{drive_id}/items/{item_id}/children"     else:         url = f"https://graph.microsoft.com/v1.0/drives/{drive_id}/root/children"     while url:         r = requests.get(url, headers=headers)         r.raise_for_status()         data = r.json()         for item in data.get('value', []):             yield item         url = data.get('@odata.nextLink') def walk_drive(drive_id, parent_id=None, path=""):     for item in list_children(drive_id, parent_id):         if item['folder'] if 'folder' in item else False:             new_path = os.path.join(path, item['name'])             walk_drive(drive_id, item['id'], new_path)         else:             rel_path = os.path.join(path, item['name'])             yield item, rel_path # Download files that changed (compare by eTag or lastModifiedDateTime) for item, rel_path in walk_drive(DRIVE_ID):     dest = os.path.join(STAGING, rel_path)     os.makedirs(os.path.dirname(dest), exist_ok=True)     # Use drive item content endpoint     download_url = f"https://graph.microsoft.com/v1.0/drives/{DRIVE_ID}/items/{item['id']}/content"     # Optionally check eTag to skip unchanged files     if not os.path.exists(dest) or item.get('eTag') != None and open(dest, 'rb').read(0) != item.get('eTag'):         r = requests.get(download_url, headers=headers, stream=True)         r.raise_for_status()         with open(dest, 'wb') as f:             for chunk in r.iter_content(32768):                 if chunk:                     f.write(chunk) 

Then rsync to target as in the PowerShell example.

Notes:

• Use item[‘eTag’] or lastModifiedDateTime for change detection; store an index file with metadata to compare across runs.

---

Example 3 — Mounting via WebDAV and using rsync

On Linux you can mount SharePoint via davfs2 and run rsync directly, but expect limitations:

1. Install davfs2
2. Add the site to /etc/fstab or mount manually
3. Run rsync

Mount example:

sudo apt-get install davfs2 mkdir -p /mnt/sharepoint sudo mount -t davfs https://contoso.sharepoint.com/sites/Team/Shared%20Documents/ /mnt/sharepoint # then rsync -av --delete /mnt/sharepoint/ user@linuxhost:/var/www/sharepoint-mirror/ 

Caveats:

• WebDAV mounts can be flaky, may not expose all metadata, and may have performance issues for many small files.

---

Handling deletions and conflicts

• To mirror deletions, use rsync –delete on the final sync step and ensure staging only contains current SharePoint files.
• Maintain a local index (JSON or SQLite) keyed by file path with eTag/lastModified; compare each run to detect added/modified/deleted files and to avoid unnecessary downloads.
• For two-way sync (bi-directional), conflict resolution rules are needed (e.g., newest-wins, or keep SharePoint authoritative). Two-way sync is complex and may require transaction logging.

---

Scheduling and reliability

• On Linux/macOS: use cron, systemd timers, or Kubernetes jobs for scheduled runs.
• On Windows: Task Scheduler.
• Implement retries for transient network errors, exponential backoff for API rate limits, and logging/alerting on failures.
• Consider chunked downloads/uploads for very large files.

---

Performance tips

• Paginate API requests and parallelize file downloads (respect rate limits).
• Use compression on the rsync leg: rsync -avz for WAN transfers.
• Skip unchanged files using metadata checks to avoid re-downloading large files unnecessarily.
• For extremely large libraries, consider incremental runs and sharding by folder.

---

Security considerations

• Use app-only OAuth with least-privilege permissions.
• Store client secrets/certs securely (Key Vault, environment vars with limited access).
• Use HTTPS for all transfers and harden the target host.
• Monitor and rotate credentials periodically.

---

Example workflow summary (end-to-end)

1. Authenticate via app-only OAuth to Microsoft Graph.
2. Enumerate files in the SharePoint document library and collect metadata (path, id, eTag, lastModified).
3. Compare metadata with a cached index to detect changes.
4. Download new/changed files to a local staging directory.
5. Rsync staging directory to the final UNIX target with –delete to mirror deletions.
6. Update the local index with new metadata and log the run.

---

Troubleshooting common issues

• ⁄₄₀₃ errors: check permissions and token scopes.
• Timeouts: increase HTTP timeouts and paginate downloads.
• Incorrect folder structure: ensure you reconstruct paths from FileRef or Graph path segments.
• WebDAV errors: prefer API-based approaches for reliability.

---

Closing notes

Automating SharePoint rsync list tasks is a practical way to bridge SharePoint document libraries with Unix-style hosts. For robust production systems, favor Microsoft Graph API with proper authentication, maintain a metadata index for efficient delta transfers, and separate staging from the final rsync step to take advantage of rsync’s efficient transfer capabilities. The templates above can be adapted into CI/CD pipelines, systemd services, or scheduled jobs to create a resilient sync pipeline.