> For the complete documentation index, see [llms.txt](https://karini-ai.gitbook.io/karini-ai-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://karini-ai.gitbook.io/karini-ai-documentation/recipes/knowledgebase-recipe/data-storage-connectors/amazon-s3-manifest.md).

# Amazon S3 manifest

This is a sample manifest, showcasing the structure with references to documents stored in Amazon S3 and associated metadata for each entry.

1. **Manifest file with `source_ref` and `metadata`**

```
[
  {
    "source_ref": "s3://examplebucket/blogs/example.com-How to Optimize AWS Lambda for Scalability.pdf",
    "metadata": {
      "user_name": "Alice Cooper"
    }
  },
  {
    "source_ref": "s3://examplebucket/blogs/example.com-Introduction to Amazon S3 Glacier for Data Archiving.pdf",
    "metadata": {
      "user_name": "Bob Johnson"
    }
  },
  {
    "source_ref": "s3://examplebucket/blogs/example.com-AWS EC2 Instance Types and Choosing the Right One.pdf",
    "metadata": {
      "user_name": "Charlie Daniels"
    }
  },
  {
    "source_ref": "s3://examplebucket/blogs/example.com-Amazon DynamoDB: Best Practices and Common Pitfalls.pdf",
    "metadata": {
      "user_name": "Diana Smith"
    }
  },
  {
    "source_ref": "s3://examplebucket/blogs/example.com-Deep Dive into Amazon RDS Performance Tuning.pdf",
    "metadata": {
      "user_name": "Edward White"
    }
  },
  {
    "source_ref": "s3://examplebucket/blogs/example.com-Amazon ElasticSearch Service: What You Need to Know.pdf",
    "metadata": {
      "user_name": "Fiona Green"
    }
  },
  {
    "source_ref": "s3://examplebucket/blogs/example.com-Optimizing CloudWatch Monitoring for AWS Environments.pdf",
    "metadata": {
      "user_name": "George Harris"
    }
  }
]

```

* **`source_ref`**: A reference to the file stored in an S3 bucket, typically a URL or path pointing to the file.
* **`metadata`**: An object containing metadata information, such as the user's name who uploaded the file or is associated with it.
  * **`user_name`**: The name of the user associated with this particular document or data entry.

2. **Manifest file with `sorce_ref`, `processed_source_ref` and `metadata`**

```
[
  {
    "source_ref": "s3://examplebucket/resumes/JohnDoe_Resume.pdf",
    "processed_source_ref": "s3://examplebucket/resumes/jsonlfiles/JohnDoe_Resume_123456.json",
    "metadata": {
      "name": "John Doe",
      "experience": "6y_2m",
      "source": "cybersecurity-analyst-resumes"
    }
  },
  {
    "source_ref": "s3://examplebucket/resumes/JaneSmith_Resume.pdf",
    "processed_source_ref": "s3://examplebucket/resumes/jsonlfiles/JaneSmith_Resume_654321.json",
    "metadata": {
      "name": "Jane Smith",
      "experience": "4y_8m",
      "source": "cybersecurity-analyst-resumes"
    }
  },
  {
    "source_ref": "s3://examplebucket/resumes/MichaelBrown_Resume.pdf",
    "processed_source_ref": "s3://examplebucket/resumes/jsonlfiles/MichaelBrown_Resume_789123.json",
    "metadata": {
      "name": "Michael Brown",
      "experience": "7y_5m",
      "source": "cybersecurity-analyst-resumes"
    }
  },
  {
    "source_ref": "s3://examplebucket/resumes/SusanWhite_Resume.pdf",
    "processed_source_ref": "s3://examplebucket/resumes/jsonlfiles/SusanWhite_Resume_321654.json",
    "metadata": {
      "name": "Susan White",
      "experience": "9y_3m",
      "source": "cybersecurity-analyst-resumes"
    }
  },
  {
    "source_ref": "s3://examplebucket/resumes/PaulGreen_Resume.pdf",
    "processed_source_ref": "s3://examplebucket/resumes/jsonlfiles/PaulGreen_Resume_987654.json",
    "metadata": {
      "name": "Paul Green",
      "experience": "5y_0m",
      "source": "cybersecurity-analyst-resumes"
    }
  }
]

```

* **`source_ref`**: The S3 path or URL to the original unprocessed document.
* **`processed_source_ref`**: The S3 path or URL to the processed version of the document (usually in JSON format).
* **`metadata`**: Contains additional information about the document such as the user's name, experience, and the source of the document.
  * **`name`**: The name of the person associated with the document.
  * **`experience`**: The professional experience of the person, typically in "years\_months" format.
  * **`source`**: The category or source of the document (e.g., "cybersecurity-analyst-resumes").