S3

​

Setup Overview

​

Authentication

​

How to configure and enable the S3 Connector

​

Step 1: Create an IAM User in AWS

1. Sign in to AWS Management Console:
   Navigate to console.aws.amazon.com and sign in with your AWS account credentials.
2. Access IAM Service:
   • In the AWS Management Console, search for “IAM” in the top search bar
   • Click on “IAM” from the results to open the Identity and Access Management console

3. Navigate to Users:
   • In the left sidebar, click “Users”
   • Click the “Create user” button (orange button at the top right)

​

Step 2: Configure User Details

1. Enter User Name:
   • Enter a descriptive name for the user (e.g., “PipesHub-S3-Connector”)
   • Click “Next” button

2. Set Permissions:
   • Select “Attach policies directly”
   • Search for and select “AmazonS3ReadOnlyAccess” policy
   • This policy provides read-only access to all S3 buckets

3. Review and Create:
   • Review the user configuration
   • Click “Create user” button at the bottom

​

Step 3: Create Access Keys

1. Access Security Credentials:
   • After creating the user, you’ll be redirected to the user details page
   • Click on the “Security credentials” tab
   • Scroll down to the “Access keys” section

2. Create Access Key:
   • Click “Create access key” button
   • Select “Application running outside AWS” as the use case
   • Check the confirmation box
   • Click “Next” button

3. Download or Copy Credentials:
   • Access Key ID: Copy this value immediately
   • Secret Access Key: Click “Show” button and copy this value immediately

4. Download Credentials (Optional):
   • Click “Download .csv file” button to save credentials to a secure location
   • Click “Done” button to complete the process

​

Step 4: Open S3 in PipesHub

1. Open the PipesHub app.
2. In the left sidebar, click Connectors (under Workspace).
3. Find the S3 tile.
4. Click + Setup on the tile.

​

Step 5: Fill in your S3 credentials

1. Instance name — type any name you like (for example S3). This is just a label so you can recognise it later.
2. Access Key — paste the Access Key ID you copied from AWS in Step 3.
3. Secret Key — paste the Secret Access Key you copied from AWS in Step 3.
4. Click Next → (bottom right).

​

Step 6: Choose how often S3 should sync

1. Sync Strategy — pick one:
   • Scheduled — PipesHub checks S3 for new or changed files automatically. (Recommended.)
   • Manual — nothing is pulled until you click Sync yourself.
2. Sync Interval — only shown if you chose Scheduled. Pick how often to check S3 (for example 1 Hour).

​

Step 7: Pick what to sync (filters)

1. Enable Manual Indexing — leave this off. (Turning it on stops PipesHub from automatically indexing new files for search.)
2. Click + Add filter to add a filter. You’ll see four options:
   • Bucket Names — pick which S3 buckets to include or exclude. PipesHub loads your bucket list automatically — just select from the dropdown.
   • File Extensions — include or exclude specific file types. Type them comma-separated, e.g. pdf, docx, txt.
   • Modified Date — only sync files changed after / before / between the dates you choose.
   • Created Date — same idea, but based on when the file was first uploaded to S3.
3. Click Save Configuration (bottom right) when you’re done.

​

Step 8: Turn on sync

1. Find the Sync Enabled row at the bottom of the card.
2. Click the toggle so it turns on.

​

Step 9: Check that sync is working

• Records Status — the main scoreboard:
  • Total — how many files PipesHub knows about.
  • Failed — files that couldn’t be fetched (usually a permission problem in AWS).
  • Unsupported — file types PipesHub can’t read (e.g. encrypted or unknown binary).
  • Processing — being indexed right now.
  • Not Started — waiting in the queue.
• Records by Type — shows how many FILE records have been synced.
• Sync (top right) — click it to pull the latest changes right now.
• Full sync — click this to re-scan everything from scratch (slower; use only if something looks wrong).
• Manage Configuration — opens the setup panel again if you need to change keys or filters.

​

Advanced: Custom IAM Policy

​

Example Custom Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets",
        "s3:GetBucketLocation"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

​

Enhanced Custom Policy (Multiple Buckets)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:GetObjectVersion",
                "s3:GetObjectAttributes",
                "s3:GetObjectTagging"
            ],
            "Resource": [
                "arn:aws:s3:::*/*",
                "arn:aws:s3:::your-bucket-name-1",
                "arn:aws:s3:::your-bucket-name-2",
                "arn:aws:s3:::your-bucket-name-3"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*"
        }
    ]
}

1. In IAM, go to “Policies” → “Create policy”
2. Use the JSON editor to paste the policy above (replace your-bucket-name with your actual bucket name)
3. Name the policy (e.g., “PipesHub-S3-ReadOnly-SpecificBuckets”)
4. Attach the policy to your IAM user instead of the managed policy

​

Supported Features

• Buckets: All accessible buckets in your AWS account
• Files: All file types with their metadata (name, size, modification date, MIME type)
• Folders: Directory structure and organization using S3 prefixes
• Permissions: Access control based on bucket and object-level permissions

​

Data Sync Behavior

​

Initial Sync

• Bucket Discovery: Lists all accessible buckets in your AWS account (or uses configured bucket filters)
• Region Detection: Automatically detects and caches the region for each bucket
• Full Object Listing: Fetches all objects from selected buckets using pagination
• Metadata Extraction: Extracts file names, sizes, modification dates, MIME types, and paths
• Folder Structure: Creates logical folder structure from S3 object prefixes
• Indexing: Indexes file content for AI-powered search (based on indexing filter settings)
• Permission Assignment: Assigns permissions based on connector scope (Personal or Team)

​

Incremental Sync

• Timestamp-Based Detection: Uses LastModified timestamps to identify changed objects
• Sync Point Tracking: Stores the last sync timestamp per bucket in sync points
• Change Detection: Only fetches objects modified since the last sync timestamp
• Pagination Resume: Uses continuation tokens to resume interrupted syncs
• Efficient Updates: Processes only new, modified, or deleted objects
• API Optimization: Reduces API calls by 90%+ compared to full syncs

1. Connector reads the sync point for each bucket (stored in our database)
2. Retrieves the last_sync_time timestamp
3. Queries S3 for objects with LastModified >= last_sync_time
4. Processes only the changed objects
5. Updates the sync point with the new maximum timestamp

​

Permission Handling

• IAM-Based Access: Only syncs buckets and objects the IAM user has permission to access
• Bucket Policies: Respects S3 bucket policies and access control lists (ACLs)
• PipesHub Scope:
  • Personal Scope: Only the connector creator can access synced content
  • Team Scope: All organization users have read access to synced content
• Permission Inheritance: Files inherit permissions from their parent bucket (Record Group)
• Access Control: Users see only content they have permission to view in both AWS and PipesHub

​

Sync Frequency

• Default interval: 60 minutes
• Configurable: 15 min, 30 min, 1 hour, 4 hours, 24 hours
• Runs automatically in the background
• Best for: Keeping data up-to-date with minimal manual intervention

• Triggered on-demand from the connector settings
• Best for: Testing, troubleshooting, or syncing after bulk changes
• Useful when: You need immediate sync after uploading many files