kosli snapshot s3

Synopsis

kosli snapshot s3 ENVIRONMENT-NAME [flags]

Report a snapshot of the content of an AWS S3 bucket to Kosli. To authenticate to AWS, you can either:

provide the AWS static credentials via flags or by exporting the equivalent KOSLI env vars (e.g. KOSLI_AWS_KEY_ID)
export the AWS env vars (e.g. AWS_ACCESS_KEY_ID).
Use a shared config/credentials file under the $HOME/.aws

Option 1 takes highest precedence, while option 3 is the lowest. More details can be found here: https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/#specifying-credentials You can report the entire bucket content, or filter some of the content using --include / --exclude (literal prefix match) or --include-regex / --exclude-regex (Go regular expressions matched against the full object key). In all cases, the content is reported as one artifact. If you wish to report separate files/dirs within the same bucket as separate artifacts, you need to run the command twice. Object keys are never used as local file names: by default each object is downloaded to a temporary file, hashed and removed, and the fingerprint is computed from the keys and the content digests, so any key S3 accepts can be fingerprinted on any operating system. Keys that cannot form a directory tree are rejected and fail the snapshot, naming every key involved: a key containing a .. segment, two keys that resolve to the same path (such as a//b and a/b), or an object whose key is also a prefix of other objects (such as a beside a/b). A legitimate key of that shape can be left out with --exclude-regex (anchor and escape it, since the pattern is a regular expression matched against the whole key); when --include or --include-regex is set, exclude filters are ignored, so narrow the include filter instead. By default each object’s SHA256 comes from downloading the object and hashing it. --fingerprint-source metadata reads the SHA256 checksum S3 stores for the object instead, which skips the download, the temporary disk and the hashing. Everything else, like the keys, the .kosli_ignore rules, the way digests combine into the fingerprint, is the same in both modes, so the fingerprint is identical and a snapshot matches the artifact you attested either way. Two conditions apply:

Every contributing object must carry a full-object SHA256 checksum. S3 only stores one when the upload asked for it, for example aws s3api put-object --checksum-algorithm SHA256. Objects without one fail the snapshot, all named in one run.
A multipart upload gets a composite SHA256, which hashes the checksums of the parts rather than the object content, so it cannot serve as the object’s fingerprint. Such an object can be collapsed into a single part in place with aws s3api copy-object --checksum-algorithm SHA256 --copy-source yourBucket/yourKey --bucket yourBucket --key yourKey. A root .kosli_ignore is still downloaded in this mode, because its rules decide which objects contribute; the objects it excludes are never fetched and need no checksum. Reading a checksum does not need fewer permissions than downloading: AWS requires s3:GetObject for both, and an SSE-KMS encrypted object additionally needs kms:GenerateDataKey and kms:Decrypt either way.

To specify paths in a directory artifact that should always be excluded from the SHA256 calculation, you can add a .kosli_ignore file to the root of the artifact. Each line should specify a relative path or path glob to be ignored. You can include comments in this file, using #. The .kosli_ignore file is always treated as part of the artifact: its own entries cannot exclude it, so the exclusion list cannot be changed without changing the fingerprint. Paths the list already matches stay excluded whatever is later added there, so keep its entries as narrow as possible.

Flags

Flag	Type	Description
`--aws-key-id`	string	The AWS access key ID.
`--aws-region`	string	The AWS region.
`--aws-secret-key`	string	The AWS secret access key.
`--bucket`	string	The name of the S3 bucket.
`--download-budget`	string	[optional] The maximum total size of the S3 objects downloading at the same time, which caps the temporary disk the snapshot uses. A bare number is megabytes; add K, M, G or T (optionally followed by B) to choose the unit, e.g. 512M or 8G. An object larger than the budget still downloads, on its own. Objects are downloaded to the OS temporary directory. Has no effect with `--fingerprint-source` metadata, which reads stored checksums and uses no temporary disk. (default “512M”)
`--download-concurrency`	int	[optional] The number of S3 objects to fetch at the same time when fingerprinting the bucket. When downloading, the default, each object in flight may hold up to 40 MB of download buffers in memory, on top of the disk the `--download-budget` allows. With `--fingerprint-source` metadata it bounds the checksum reads instead, which hold neither, and defaults to 32. (default 8)
`-D`, `--dry-run`	bool	[optional] Run in dry-run mode. When enabled, no data is sent to Kosli and the CLI exits with 0 exit code regardless of any errors.
`-x`, `--exclude`	strings	[optional] The comma separated list of file and/or directory paths in the S3 bucket to exclude when fingerprinting. Paths match by literal prefix. Cannot be used together with `--include` or `--include-regex`.
`--exclude-regex`	strings	[optional] The comma separated list of Go regular expressions matched against object keys in the S3 bucket to exclude when fingerprinting. Cannot be used together with `--include` or `--include-regex`.
`--fingerprint-source`	string	[defaulted] Where each object’s SHA256 comes from when fingerprinting the bucket. Valid sources are: [content, metadata]. ‘content’ downloads every contributing object and hashes it. ‘metadata’ reads the SHA256 checksum S3 stores for each object instead, which skips the download but requires every contributing object to have been uploaded with a full-object SHA256 checksum. Both produce the same fingerprint and need the same permissions. (default “content”)
`-h`, `--help`	bool	help for s3
`-i`, `--include`	strings	[optional] The comma separated list of file and/or directory paths in the S3 bucket to include when fingerprinting. Paths match by literal prefix. Cannot be used together with `--exclude` or `--exclude-regex`.
`--include-regex`	strings	[optional] The comma separated list of Go regular expressions matched against object keys in the S3 bucket to include when fingerprinting. Cannot be used together with `--exclude` or `--exclude-regex`.

Flags inherited from parent commands

Flag	Type	Description
`-a`, `--api-token`	string	The Kosli API token.
`-A`, `--auto-environment`	bool	[optional] Create the environment (with the type inferred from the snapshot subcommand) if it does not already exist, before reporting the snapshot.
`-c`, `--config-file`	string	[optional] The Kosli config file path. Config is read from this path or the default only, never implicitly from the current directory. (default “$HOME/.kosli.yml”)
`--debug`	bool	[optional] Print debug logs to stdout.
`--environment-description`	string	[optional] The environment description.
`--exclude-scaling`	bool	[optional] Exclude scaling events for snapshots. Snapshots with scaling changes will not result in new environment records. (DEPRECATED: this flag is deprecated and will be removed in a future version. Scaling events do not trigger new snapshots.)
`-H`, `--host`	string	[defaulted] The Kosli endpoint. (default “https://app.kosli.com”)
`--http-proxy`	string	[optional] The HTTP proxy URL including protocol and port number. e.g. `http://proxy-server-ip:proxy-port`
`--include-scaling`	bool	[optional] Include scaling events for snapshots. Snapshots with scaling changes will result in new environment records. (DEPRECATED: this flag is deprecated and will be removed in a future version. Scaling events do not trigger new snapshots.)
`-r`, `--max-api-retries`	int	[defaulted] How many times should API calls be retried when the API host is not reachable. (default 3)
`--org`	string	The Kosli organization.
`-q`, `--quiet`	bool	[optional] Suppress non-critical warning messages. Errors and normal output are not affected. If both `--quiet` and `--debug` are set, `--debug` wins.

Examples Use Cases

These examples all assume that the flags --api-token, --org, --host, (and --flow, --trail when required), are set/provided.

report the contents of an entire AWS S3 bucket (AWS auth provided in env variables)

export AWS_REGION=yourAWSRegion
export AWS_ACCESS_KEY_ID=yourAWSAccessKeyID
export AWS_SECRET_ACCESS_KEY=yourAWSSecretAccessKey

kosli snapshot s3 yourEnvironmentName 
	--bucket yourBucketName 

report what is running in an AWS S3 bucket (AWS auth provided in flags)

kosli snapshot s3 yourEnvironmentName 
	--bucket yourBucketName 
	--aws-key-id yourAWSAccessKeyID 
	--aws-secret-key yourAWSSecretAccessKey 
	--aws-region yourAWSRegion 

report a subset of contents of an AWS S3 bucket (AWS auth provided in env variables)

export AWS_REGION=yourAWSRegion
export AWS_ACCESS_KEY_ID=yourAWSAccessKeyID
export AWS_SECRET_ACCESS_KEY=yourAWSSecretAccessKey

kosli snapshot s3 yourEnvironmentName 
	--bucket yourBucketName 
	--include file.txt,path/within/bucket 

report contents of an entire AWS S3 bucket, except for some paths (AWS auth provided in env variables)

export AWS_REGION=yourAWSRegion
export AWS_ACCESS_KEY_ID=yourAWSAccessKeyID
export AWS_SECRET_ACCESS_KEY=yourAWSSecretAccessKey

kosli snapshot s3 yourEnvironmentName 
	--bucket yourBucketName 
	--exclude file.txt,path/within/bucket 

report contents of an AWS S3 bucket, excluding all PNG files via a regex

kosli snapshot s3 yourEnvironmentName 
	--bucket yourBucketName 
	--exclude-regex '.*\.png$' 

report contents of an AWS S3 bucket without downloading the objects,

using the SHA256 checksums S3 stores for them

kosli snapshot s3 yourEnvironmentName 
	--bucket yourBucketName 
	--fingerprint-source metadata 

General

Top-level commands

kosli allow

kosli archive

kosli assert

kosli attest

kosli begin

kosli create

kosli delete

kosli diff

kosli disable / enable

kosli evaluate

kosli get

kosli join

kosli list

kosli log

kosli rename

kosli report

kosli rotate

kosli snapshot

kosli unarchive

kosli update

Synopsis

Flags

Flags inherited from parent commands

Examples Use Cases

​Synopsis

​Flags

​Flags inherited from parent commands

​Examples Use Cases

Synopsis

Flags

Flags inherited from parent commands

Examples Use Cases