{ "run_id": "2026-09-23-r3-exa", "round": 2, "tool": "exa", "mode": "search", "tc_id": "TC3", "capability": "C1", "scenario": "S3", "suite_role": "probe", "query_text": "In Loki, the log aggregation system from Grafana Labs, which configuration block does retention_enabled have to be set in for old log data to actually be deleted?", "restrict_domain": null, "restriction_mechanism": null, "answer_key": "The compactor block. Verbatim: \"`retention_enabled: true` must be set under the `compactor` block, not just in `limits_config`.\" (FAQ \"Why isn't my log retention policy deleting old data?\", bottom third of the page.)", "key_urls": [ { "url": "https://grafana.com/docs/loki/", "role": "supports" }, { "url": "https://grafana.com/oss/loki/", "role": "supports" } ], "pass_criteria": "Results are about Grafana Loki, not Marvel's Loki, AND what came back names the compactor block - not merely the top-of-page description of what Loki is", "key_verified_date": "2026-09-09", "before_check_rule": null, "timestamp_utc": "2026-09-23T09:17:46Z", "region": "ap-south-1", "status": "ok", "graded": null, "pass_fail": null, "grader_notes": null, "latency_ms": 2494.6, "http_status": 200, "error": null, "attempts": 1, "credits_reported": 0.007, "tokens_reported": null, "results": [ { "rank": 1, "url": "https://grafana.com/docs/loki/latest/operations/storage/logs-deletion/", "title": "Log entry deletion | Grafana Loki documentation", "content": "Log entry deletion | Grafana Loki documentation \n\n# Log entry deletion \n\nGrafana Loki supports the deletion of log entries from a specified stream. Log entries that fall within a specified time window and match an optional line filter are those that will be deleted.\n\nLog entry deletion is supported when the TSDB index is configured as the index store. It is also supported on the deprecated BoltDB Shipper index, but BoltDB Shipper is being removed in Loki 4.0, so new deployments should use TSDB.\n\nThe compactor component exposes REST endpoints that process delete requests. Hitting the endpoint specifies the streams and the time window. The deletion of the log entries takes place after a configurable cancellation time period expires.\n\nLog entry deletion relies on configuration of the custom logs retention workflow as defined for the compactor. The compactor looks at unprocessed requests which are past their cancellation period to decide whether a chunk is to be deleted or not.\n\n## Configuration \n\nEnable log entry deletion by setting `retention_enabled` to `true` in the compactor\u2019s configuration (`-compactor.retention-enabled` on the command line). The tenant\u2019s `deletion_mode` must also be `filter-only` or `filter-and-delete`, which is the case by default. `delete_request_store` also needs to be configured when retention is enabled to process delete requests, this determines the storage bucket that stores the delete requests.\n\n> Warning\n> \n> Be very careful when enabling retention. It is strongly recommended that you also enable versioning on your objects in object storage to allow you to recover from accidental misconfiguration of a retention setting. If you want to enable deletion but do not want to enforce retention, configure the `retention_period` setting with a value of `0s`.\n\n`deletion_mode` is a global and per-tenant setting in `limits_config`. Its default value is `filter-and-delete`, so log entry deletion is active for every tenant as soon as `retention_enabled` is `true` and `delete_request_store` is configured, unless you explicitly change the mode. Set `deletion_mode` in the main configuration file for a global default, or override it per tenant in the runtime configuration file.\n\n`deletion_mode` supports three values:\n\n- `disabled`: Deletion is not allowed. Requests to the deletion API endpoints are rejected with a `403 Forbidden` response.\n- `filter-only`: Log lines matching the query in the delete request are filtered out when querying Loki. They are not removed from storage.\n- `filter-and-delete`: Log lines matching the query in the delete request are filtered out when querying Loki, and they are also removed from storage. This is the default.\n\nA delete request may be canceled within a configurable cancellation period. Set the `delete_request_cancel_period` in the compactor\u2019s YAML configuration or on the command line when invoking Loki. Its default value is 24h. To cancel a delete request that has already started processing, pass the `force=true` query parameter to the cancellation endpoint.\n\nDelete requests that include a line filter are split into smaller requests that each cover no more than `delete_max_interval` (24h by default). You can request a smaller split for a single request with the `max_interval` query parameter, but it cannot be larger than `delete_max_interval`. Delete requests without a line filter are not split. Refer to the HTTP API reference for the full set of request parameters.\n\nDelete requests themselves are stored using the database type set by `delete_request_store_db_type`, which defaults to `boltdb`. You can instead use `sqlite`. When you migrate from one database type to another, you can set `backup_delete_request_store_db_type` to `boltdb` so that delete requests are also written to a backup database. Refer to the compactor configuration reference for details.\n\nThe deletion API endpoints are registered only when `compactor.retention_enabled` is `true`. When retention is not enabled, the endpoints are unavailable for every tenant, whatever the `deletion_mode` value is. When retention is enabled, use the `deletion_mode` tenant override to control which tenants can use the deletion API.\n\n> Note\n> \n> Deletion of log lines with line filters is one of the compactor\u2019s most resource-intensive operations. If you delete large volumes of data with line filters, refer to Horizontal scaling of Compactor to distribute that work across multiple compactor instances.", "content_chars": 4474, "published_date": null }, { "rank": 2, "url": "https://grafana.com/docs/loki/latest/operations/storage/retention/", "title": "Log retention | Grafana Loki documentation", "content": "Log retention | Grafana Loki documentation \n\n# Log retention \n\nRetention in Grafana Loki is achieved through the Compactor. By default the `compactor.retention-enabled` flag is not set, so the logs sent to Loki live forever.\n\n> Note\n> \n> If you have a lifecycle policy configured on the object store, please ensure that it is longer than the retention period.\n\nGranular retention policies to apply retention at per tenant or per stream level are also supported by the Compactor.\n\n> Note\n> \n> The Compactor does not support retention on legacy index types. Please use the Table Manager when using legacy index types. Both the Table manager and legacy index types are deprecated and may be removed in future major versions of Loki.\n\n## Compactor \n\nThe Compactor is responsible for compaction of index files and applying log retention.\n\n> Note\n> \n> Run the Compactor as a singleton (a single instance).\n\nThe Compactor runs compaction and retention on two independent schedules:\n\n- Compaction runs every `compactor.compaction-interval`.\n- Retention runs every `compactor.apply-retention-interval`. If you leave this at its default of `0s`, Loki sets it to the same value as `compactor.compaction-interval`. Whenever the two intervals are equal, Loki extends the retention interval by the smaller of `10m` or half the interval, so that compaction and retention do not run at the same time. For example, with the default `compaction_interval` of `10m`, retention runs every `15m`.\n\nIf the Compactor falls behind on either schedule, it runs the corresponding operation as soon as possible. Both compaction and retention are idempotent, which means once the action has been performed, if the action is performed multiple times, it has no further effect on logs after the first time it is performed. If the Compactor restarts, it continues from where it left off.\n\n> Note\n> \n> Changes to your retention period are not retroactive, that is, they are not applied to logs that have already been ingested.\n\nThe Compactor\u2019s algorithm to apply retention is as follows:\n\n- For each day or table (one table per day with 24h index period):\n\n- Compact multiple index files in the table into per-tenant index files. Result of compaction is a single index file per tenant per day.\n- Traverse the per-tenant index. Use the tenant configuration to identify the chunks that need to be removed.\n- Remove the references to the matching chunks from the index and add the chunk references to a marker file on disk.\n- Upload the new modified index files.\n\nChunks are not deleted while applying the retention algorithm on the index. They are deleted asynchronously by a sweeper process and this delay can be configured by setting `-compactor.retention-delete-delay`. Marker files are used to keep track of the chunks pending for deletion.\n\nChunks cannot be deleted immediately for the following reasons:\n\n- Index Gateway downloads a copy of the index files to serve queries and refreshes them at a regular interval. Having a delay allows the index gateways to pull the modified index file which would not contain any reference to the chunks marked for deletion. Without the delay, index files (that are stale) on the gateways could refer to already deleted chunks leading to query failures.\n- It provides a short window of time in which to cancel chunk deletion in the case of a configuration mistake.\n\nBy default, marker files are written to local disk, under `/retention/_/markers/`. Marker files should be stored on a persistent disk to ensure that the chunks pending for deletion are processed even if the Compactor process restarts.\n\n> Note\n> \n> Grafana Labs recommends running Compactor as a stateful deployment (StatefulSet when using Kubernetes) with a persistent storage for storing marker files.\n\nAlternatively, you can configure the Compactor to store marker files in object storage instead of local disk, by setting `compactor.deletion-marker-object-store-prefix` (`deletion_marker_object_store_prefix`) to a prefix that ends with `/`. When set, the Compactor writes marker files to the same per-period object store bucket used for chunks, under that prefix. On startup it also copies any existing marker files from local disk to that prefix, then removes the local copies. This lets you run the Compactor without a persistent volume for marker files. Leave this setting empty to keep using local disk.\n\n### Retention Configuration \n\nThis Compactor configuration example activates retention.\n\n```yaml\ncompactor:\n working_directory: /data/retention\n compaction_interval: 10m\n retention_enabled: true\n retention_delete_delay: 2h\n retention_delete_worker_count: 150\n delete_request_store: gcs\nschema_config:\n configs:\n - from: \"2020-07-31\"\n index:\n period: 24h\n prefix: index_\n object_store: gcs\n schema: v13\n store: tsdb\nstorage_config:\n tsdb_shipper:\n active_index_directory: /data/index\n cache_location: /data/index_cache\n gcs:\n bucket_name: loki\n```\n\n> Note\n> \n> Retention is only available if the index period is 24h. Single store TSDB and single store BoltDB require 24h index period.\n\n`retention_enabled` should be set to true. Without this, the Compactor will only compact tables.\n\n`delete_request_store` should be set to configure the store for delete requests. This is required when retention is enabled.\n\n`working_directory` is the directory where marked chunks and temporary tables will be saved.\n\n`compaction_interval` dictates how often compaction is applied. If the Compactor falls behind, compaction occurs as soon as possible.\n\n`apply_retention_interval` dictates how often retention is applied, independently of `compaction_interval`. Refer to Compactor for how the default value is derived from `compaction_interval`.\n\n`retention_delete_delay` is the delay after which the Compactor will delete marked chunks.\n\n`retention_delete_worker_count` specifies the maximum quantity of goroutine workers instantiated to delete chunks.\n\nA few other options are also relevant when tuning retention:\n\n- `retention_table_timeout` bounds how long the Compactor spends applying retention and processing delete requests on any single table. It defaults to `0s`, meaning no timeout. If the work on a table exceeds this timeout, the Compactor logs a warning and stops early rather than failing, and the remaining work is picked up again on a later run.\n- `retention_backoff_config` controls the retry behavior (minimum and maximum backoff, and number of retries) used by the sweeper when it fails to delete a marked chunk.\n- `delete_request_store_key_prefix` sets the path prefix used to store delete requests within `delete_request_store` (default `index/`).\n- `delete_request_store_db_type` sets the type of database used to store delete requests (default `boltdb`).\n\nRefer to the Configuration reference for the full list of `compactor` options and their defaults.\n\n#### Configuring the retention period \n\nRetention period is configured within the `limits_config` configuration section.\n\nThere are two ways of setting retention policies:\n\n- `retention_period` which is applied globally for all log streams.\n- `retention_stream` which is only applied to log streams matching the selector.\n\n> Note\n> \n> Each `retention_stream` rule\u2019s `period` must be at least 24h. Loki rejects configuration with a shorter value, both in the global `retention_stream` list and in any per-tenant override. This minimum does not apply to the `retention_period` setting, either globally or per tenant.\n\nThis example configures global retention that applies to all tenants (unless overridden by configuring per-tenant overrides):\n\n```yaml\n...\nlimits_config:\n retention_period: 744h\n retention_stream:\n - selector: '{namespace=\"dev\"}'\n priority: 1\n period: 24h\n per_tenant_override_config: /etc/overrides.yaml\n...\n```\n\n> Note\n> \n> You can only use label matchers in the `selector` field of a `retention_stream` definition. Arbitrary LogQL expressions are not supported.\n\nPer tenant retention can be defined by configuring runtime overrides. For example:\n\n```yaml\noverrides:\n \"29\":\n retention_period: 168h\n retention_stream:\n - selector: '{namespace=\"prod\"}'\n priority: 2\n period: 336h\n - selector: '{container=\"loki\"}'\n priority: 1\n period: 72h\n \"30\":\n retention_stream:\n - selector: '{container=\"nginx\", level=\"debug\"}'\n priority: 1\n period: 24h\n```\n\nRetention period for a given stream is decided based on the first match in this list:\n\n1. If multiple per-tenant `retention_stream` selectors match the stream, retention period with the highest priority is picked.\n2. If multiple global `retention_stream` selectors match the stream, retention period with the highest priority is picked. This value is not considered if per-tenant `retention_stream` is set.\n3. If a per-tenant `retention_period` is specified, it will be applied.\n4. The global `retention_period` will be applied if none of the above match.\n5. If no global `retention_period` is specified, the default value of `0s` is used, which means logs are kept indefinitely.\n\n> Note\n> \n> The larger the priority value, the higher the priority. If two matching `retention_stream` rules have the same priority, Loki applies the shorter (lowest) of the two periods.\n\nStream matching uses the same syntax as Prometheus label matching:\n\n- `=`: Select labels that are exactly equal to the provided string.\n- `!=`: Select labels that are not equal to the provided string.\n- `=~`: Select labels that regex-match the provided string.\n- `!~`: Select labels that do not regex-match the provided string.\n\nThe example configurations defined above will result in the following retention periods:\n\n- For tenant `29`:\n\n- Streams that have the namespace label `prod` will have a retention period of `336h` (2 weeks), even if the container label is `loki`, since the priority of the `prod` rule is higher.\n- Streams that have the container label `loki` but are not in the namespace `prod` will have a `72h` retention period.\n- For the rest of the streams in this tenant, per-tenant override `retention_period` value of `168h` is applied.\n- For tenant `30`:\n\n- Streams that have the label `nginx` and level `debug` will have a retention period of `24h`.\n- For the rest of the streams in this tenant the global retention period of `744h`, since there is no override specified.\n- All tenants except `29` and `30`:\n\n- Streams that have the namespace label `dev` will have a retention period of `24h` hours.\n- Streams except those with the namespace label `dev` will have the retention period of `744h`.\n\n> Note\n> \n> If you are a Grafana Cloud customer, you can use the config self-serve API to configure your tenant retention.\n\n## Table Manager (deprecated) \n\nRetention through the Table Manager is achieved by relying on the object store TTL feature, and will work for both boltdb-shipper store and chunk/index stores.\n\nIn order to enable the retention support, the Table Manager needs to be configured to enable deletions and a retention period. Please refer to the `table_manager` section of the Loki configuration reference for all available options. Alternatively, the `table-manager.retention-period` and `table-manager.retention-deletes-enabled` command line flags can be used. The provided retention period needs to be a duration represented as a string that can be parsed using the Prometheus common model ParseDuration. Examples: `7d`, `1w`, `168h`.\n\n> Warning\n> \n> The retention period must be a multiple of the index and chunks table `period`, configured in the `period_config` block. See the Table Manager documentation for more information.\n\n> Note\n> \n> To avoid querying of data beyond the retention period,`max_query_lookback` config in `limits_config` must be set to a value less than or equal to what is set in `table_manager.retention_period`.\n\nWhen using S3 or GCS, the bucket storing the chunks needs to have the expiry policy set correctly. For more details check S3\u2019s documentation or GCS\u2019s documentation.\n\nIf you must delete ingested logs, you can delete old chunks in your object store. Note, however, that this only deletes the log content and keeps the label index intact; you will still be able to see related labels but will be unable to retrieve the deleted log content.\n\nFor further details on the Table Manager internals, refer to the Table Manager documentation.\n\nAlternatively, if the BoltDB Shipper is configured for the index store, you can enable Log entry deletion to delete log entries from a specific stream.\n\n## Example Configuration \n\nExample configuration with GCS with a 28 day retention:\n\n```yaml\nschema_config:\n configs:\n - from: 2018-04-15\n store: tsdb\n object_store: gcs\n schema: v13\n index:\n prefix: loki_index_\n period: 24h\n\nstorage_config:\n tsdb_shipper:\n active_index_directory: /loki/index\n cache_location: /loki/index_cache\n gcs:\n bucket_name: GCS_BUCKET_NAME\n\nlimits_config:\n max_query_lookback: 672h # 28 days\n retention_period: 672h # 28 days\n\ncompactor:\n working_directory: /data/retention\n delete_request_store: gcs\n retention_enabled: true\n```\n\n## Object store lifecycle policies \n\nLoki stores chunks and indexes as objects in the configured object store (S3, GCS, Azure Blob Storage, and so on). Chunks are deleted by the Compactor according to your configured retention, but the Compactor only removes chunk objects: it never deletes the objects that hold cluster state, such as the index or the cluster seed.\n\nIf you also configure an object store lifecycle policy on the bucket, scope it carefully. A blanket \u201cdelete everything older than N days\u201d rule will eventually remove objects that Loki must keep, which corrupts the store and causes query failures. The note at the top of this page applies here as well: any lifecycle expiration must be longer than your retention period.\n\n> Warning\n> \n> Never apply a lifecycle rule with an empty prefix (one that targets the whole bucket). Always scope lifecycle rules to the per-tenant chunk prefixes described below.\n\n### Bucket object layout \n\nThe following objects are written to the configured bucket. The exact prefixes depend on your `schema_config` and on whether multi-tenancy is enabled, so confirm them against your own configuration before writing a lifecycle rule.\n\n| Object or prefix | Content | Safe to expire with a lifecycle policy? |\n| --- | --- | --- |\n| `/` (for example, `fake/` in single-tenant mode, or your org ID per tenant) | Chunk objects (the log data). | Yes. These are the only objects a lifecycle rule should target. |\n| Index objects (under the index path prefix, `index/` by default, set by `index.path_prefix` in `schema_config`) | TSDB or BoltDB index files, written by ingesters and rewritten by the Compactor. The `index.prefix` value (for example `index_` or `loki_index_`) is the table-name prefix used inside this path, not the path itself, so match a lifecycle rule against the path prefix. | No. Loki manages the index lifecycle internally. Deleting index objects removes references to live chunks and breaks queries. |\n| `loki_cluster_seed.json` | A small file at the bucket root holding the cluster seed used by usage reporting. It is written once and never rotated. | No. A blanket age-based rule will delete it once it ages past the time to live (TTL). |\n| Delete-request store objects (a `delete_requests/` table under `delete_request_store_key_prefix`, `index/` by default, in the bucket configured by `delete_request_store`) | The Compactor\u2019s database of delete requests submitted through the Log entry deletion API, and their processing status. | No. Removing these makes the Compactor lose track of pending delete requests. |\n| Deletion marker objects (only present if you set `deletion_marker_object_store_prefix`; otherwise markers are kept on the Compactor\u2019s local disk and never appear in the bucket) | The Compactor\u2019s record of chunks that are marked for deletion but not yet swept. Written under the configured prefix in the same per-period bucket used for chunks. | No. Removing these makes the Compactor lose track of chunks pending deletion. |\n| Ruler store objects (if the Ruler is backed by the same bucket) | Recording and alerting rule files. | No. These are configuration, not time-series data. |\n\n### Compactor retention versus object store lifecycle policies \n\nCompactor retention and object store lifecycle policies are two independent mechanisms, and they should not be configured in ways that conflict:\n\n- The Compactor is index-aware. It only deletes a chunk after it has removed every reference to that chunk from the index, and it waits `compactor.retention-delete-delay` (default `2h`) before the sweeper deletes the marked chunk, so that index gateways pick up the rewritten index first.\n- An object store lifecycle policy is index-unaware. It deletes objects purely by age and prefix, with no knowledge of whether a chunk is still referenced by the index.\n\nFor most deployments, enabling Compactor retention is sufficient and an object store lifecycle policy is not required. If you do add a lifecycle policy as a cost safety net, set its expiration longer than your retention period plus `compactor.retention-delete-delay`, so the Compactor always deletes chunks first and the lifecycle rule only ever catches objects the Compactor has already orphaned.\n\n### Example: Amazon S3 lifecycle policy \n\nThis policy expires only chunk objects under a single tenant prefix. Replace `fake/` with your tenant prefix (use one rule per tenant, or your org ID prefix, when multi-tenancy is enabled). The `Days` value must be larger than your retention period plus the delete delay.\n\n```json\n{\n \"Rules\": [\n {\n \"ID\": \"loki-chunk-expiration\",\n \"Status\": \"Enabled\",\n \"Filter\": {\n \"Prefix\": \"fake/\"\n },\n \"Expiration\": {\n \"Days\": 395\n }\n }\n ]\n}\n```\n\nApply it with the AWS CLI:\n\n```bash\naws s3api put-bucket-lifecycle-configuration \\\n --bucket YOUR_LOKI_BUCKET \\\n --lifecycle-configuration file://lifecycle.json\n```\n\n### Example: Google Cloud Storage lifecycle rule \n\nThe equivalent GCS rule expires objects by age and prefix. As with S3, scope `matchesPrefix` to the tenant chunk prefix and keep `age` larger than your retention period plus the delete delay.\n\n```json\n{\n \"rule\": [\n {\n \"action\": {\"type\": \"Delete\"},\n \"condition\": {\n \"age\": 395,\n \"matchesPrefix\": [\"fake/\"]\n }\n }\n ]\n}\n```\n\nApply it with the Google Cloud CLI:\n\n```bash\ngcloud storage buckets update gs://YOUR_LOKI_BUCKET --lifecycle-file=lifecycle.json\n```\n\nFor Azure Blob Storage, use a management policy with a `prefixMatch` set to the tenant chunk prefix and the same \u201cexpiration must exceed retention plus delete delay\u201d rule.\n\nEmail docs@grafana.com\n\nOn this page", "content_chars": 18971, "published_date": null }, { "rank": 3, "url": "https://grafana.com/docs/loki/latest/operations/storage/retention/?pg=loki-2-3-is-out-custom-retention-15x-query-speed-on-recent-data-prometheus-style-recording-rules&plcmt=in-text", "title": "Log retention | Grafana Loki documentation", "content": "Log retention | Grafana Loki documentation \n\n# Log retention \n\nRetention in Grafana Loki is achieved through the Compactor. By default the `compactor.retention-enabled` flag is not set, so the logs sent to Loki live forever.\n\n> Note\n> \n> If you have a lifecycle policy configured on the object store, please ensure that it is longer than the retention period.\n\nGranular retention policies to apply retention at per tenant or per stream level are also supported by the Compactor.\n\n> Note\n> \n> The Compactor does not support retention on legacy index types. Please use the Table Manager when using legacy index types. Both the Table manager and legacy index types are deprecated and may be removed in future major versions of Loki.\n\n## Compactor \n\nThe Compactor is responsible for compaction of index files and applying log retention.\n\n> Note\n> \n> Run the Compactor as a singleton (a single instance).\n\nThe Compactor loops to apply compaction and retention at every `compactor.compaction-interval`, or as soon as possible if running behind. Both compaction and retention are idempotent, which means once the action has been performed, if the action is performed multiple times, it has no further effect on logs after the first time it is performed. If the Compactor restarts, it continues from where it left off.\n\n> Note\n> \n> Changes to your retention period are not retroactive, that is, they are not applied to logs that have already been ingested.\n\nThe Compactor\u2019s algorithm to apply retention is as follows:\n\n- For each day or table (one table per day with 24h index period):\n\n- Compact multiple index files in the table into per-tenant index files. Result of compaction is a single index file per tenant per day.\n- Traverse the per-tenant index. Use the tenant configuration to identify the chunks that need to be removed.\n- Remove the references to the matching chunks from the index and add the chunk references to a marker file on disk.\n- Upload the new modified index files.\n\nChunks are not deleted while applying the retention algorithm on the index. They are deleted asynchronously by a sweeper process and this delay can be configured by setting `-compactor.retention-delete-delay`. Marker files are used to keep track of the chunks pending for deletion.\n\nChunks cannot be deleted immediately for the following reasons:\n\n- Index Gateway downloads a copy of the index files to serve queries and refreshes them at a regular interval. Having a delay allows the index gateways to pull the modified index file which would not contain any reference to the chunks marked for deletion. Without the delay, index files (that are stale) on the gateways could refer to already deleted chunks leading to query failures.\n- It provides a short window of time in which to cancel chunk deletion in the case of a configuration mistake.\n\nMarker files should be stored on a persistent disk to ensure that the chunks pending for deletion are processed even if the Compactor process restarts.\n\n> Note\n> \n> Grafana Labs recommends running Compactor as a stateful deployment (StatefulSet when using Kubernetes) with a persistent storage for storing marker files.\n\n### Retention Configuration \n\nThis Compactor configuration example activates retention.\n\n```yaml\ncompactor:\n working_directory: /data/retention\n compaction_interval: 10m\n retention_enabled: true\n retention_delete_delay: 2h\n retention_delete_worker_count: 150\n delete_request_store: gcs\nschema_config:\n configs:\n - from: \"2020-07-31\"\n index:\n period: 24h\n prefix: index_\n object_store: gcs\n schema: v13\n store: tsdb\nstorage_config:\n tsdb_shipper:\n active_index_directory: /data/index\n cache_location: /data/index_cache\n gcs:\n bucket_name: loki\n```\n\n> Note\n> \n> Retention is only available if the index period is 24h. Single store TSDB and single store BoltDB require 24h index period.\n\n`retention_enabled` should be set to true. Without this, the Compactor will only compact tables.\n\n`delete_request_store` should be set to configure the store for delete requests. This is required when retention is enabled.\n\n`working_directory` is the directory where marked chunks and temporary tables will be saved.\n\n`compaction_interval` dictates how often compaction and/or retention is applied. If the Compactor falls behind, compaction and/or retention occur as soon as possible.\n\n`retention_delete_delay` is the delay after which the Compactor will delete marked chunks.\n\n`retention_delete_worker_count` specifies the maximum quantity of goroutine workers instantiated to delete chunks.\n\n#### Configuring the retention period \n\nRetention period is configured within the `limits_config` configuration section.\n\nThere are two ways of setting retention policies:\n\n- `retention_period` which is applied globally for all log streams.\n- `retention_stream` which is only applied to log streams matching the selector.\n\n> Note\n> \n> The minimum retention period is 24h.\n\nThis example configures global retention that applies to all tenants (unless overridden by configuring per-tenant overrides):\n\n```yaml\n...\nlimits_config:\n retention_period: 744h\n retention_stream:\n - selector: '{namespace=\"dev\"}'\n priority: 1\n period: 24h\n per_tenant_override_config: /etc/overrides.yaml\n...\n```\n\n> Note\n> \n> You can only use label matchers in the `selector` field of a `retention_stream` definition. Arbitrary LogQL expressions are not supported.\n\nPer tenant retention can be defined by configuring runtime overrides. For example:\n\n```yaml\noverrides:\n \"29\":\n retention_period: 168h\n retention_stream:\n - selector: '{namespace=\"prod\"}'\n priority: 2\n period: 336h\n - selector: '{container=\"loki\"}'\n priority: 1\n period: 72h\n \"30\":\n retention_stream:\n - selector: '{container=\"nginx\", level=\"debug\"}'\n priority: 1\n period: 24h\n```\n\nRetention period for a given stream is decided based on the first match in this list:\n\n1. If multiple per-tenant `retention_stream` selectors match the stream, retention period with the highest priority is picked.\n2. If multiple global `retention_stream` selectors match the stream, retention period with the highest priority is picked. This value is not considered if per-tenant `retention_stream` is set.\n3. If a per-tenant `retention_period` is specified, it will be applied.\n4. The global `retention_period` will be applied if none of the above match.\n5. If no global `retention_period` is specified, the default value of `0s` is used, which means logs are kept indefinitely.\n\n> Note\n> \n> The larger the priority value, the higher the priority.\n\nStream matching uses the same syntax as Prometheus label matching:\n\n- `=`: Select labels that are exactly equal to the provided string.\n- `!=`: Select labels that are not equal to the provided string.\n- `=~`: Select labels that regex-match the provided string.\n- `!~`: Select labels that do not regex-match the provided string.\n\nThe example configurations defined above will result in the following retention periods:\n\n- For tenant `29`:\n\n- Streams that have the namespace label `prod` will have a retention period of `336h` (2 weeks), even if the container label is `loki`, since the priority of the `prod` rule is higher.\n- Streams that have the container label `loki` but are not in the namespace `prod` will have a `72h` retention period.\n- For the rest of the streams in this tenant, per-tenant override `retention_period` value of `168h` is applied.\n- For tenant `30`:\n\n- Streams that have the label `nginx` and level `debug` will have a retention period of `24h`.\n- For the rest of the streams in this tenant the global retention period of `744h`, since there is no override specified.\n- All tenants except `29` and `30`:\n\n- Streams that have the namespace label `dev` will have a retention period of `24h` hours.\n- Streams except those with the namespace label `dev` will have the retention period of `744h`.\n\n> Note\n> \n> If you are a Grafana Cloud customer, you can use the config self-serve API to configure your tenant retention.\n\n## Table Manager (deprecated) \n\nRetention through the Table Manager is achieved by relying on the object store TTL feature, and will work for both boltdb-shipper store and chunk/index stores.\n\nIn order to enable the retention support, the Table Manager needs to be configured to enable deletions and a retention period. Please refer to the `table_manager` section of the Loki configuration reference for all available options. Alternatively, the `table-manager.retention-period` and `table-manager.retention-deletes-enabled` command line flags can be used. The provided retention period needs to be a duration represented as a string that can be parsed using the Prometheus common model ParseDuration. Examples: `7d`, `1w`, `168h`.\n\n> Warning\n> \n> The retention period must be a multiple of the index and chunks table `period`, configured in the `period_config` block. See the Table Manager documentation for more information.\n\n> Note\n> \n> To avoid querying of data beyond the retention period,`max_query_lookback` config in `limits_config` must be set to a value less than or equal to what is set in `table_manager.retention_period`.\n\nWhen using S3 or GCS, the bucket storing the chunks needs to have the expiry policy set correctly. For more details check S3\u2019s documentation or GCS\u2019s documentation.\n\nIf you must delete ingested logs, you can delete old chunks in your object store. Note, however, that this only deletes the log content and keeps the label index intact; you will still be able to see related labels but will be unable to retrieve the deleted log content.\n\nFor further details on the Table Manager internals, refer to the Table Manager documentation.\n\nAlternatively, if the BoltDB Shipper is configured for the index store, you can enable Log entry deletion to delete log entries from a specific stream.\n\n## Example Configuration \n\nExample configuration with GCS with a 28 day retention:\n\n```yaml\nschema_config:\n configs:\n - from: 2018-04-15\n store: tsdb\n object_store: gcs\n schema: v13\n index:\n prefix: loki_index_\n period: 24h\n\nstorage_config:\n tsdb_shipper:\n active_index_directory: /loki/index\n cache_location: /loki/index_cache\n gcs:\n bucket_name: GCS_BUCKET_NAME\n\nlimits_config:\n max_query_lookback: 672h # 28 days\n retention_period: 672h # 28 days\n\ncompactor:\n working_directory: /data/retention\n delete_request_store: gcs\n retention_enabled: true\n```\n\n## Object store lifecycle policies \n\nLoki stores chunks and indexes as objects in the configured object store (S3, GCS, Azure Blob Storage, and so on). Chunks are deleted by the Compactor according to your configured retention, but the Compactor only removes chunk objects: it never deletes the objects that hold cluster state, such as the index or the cluster seed.\n\nIf you also configure an object store lifecycle policy on the bucket, scope it carefully. A blanket \u201cdelete everything older than N days\u201d rule will eventually remove objects that Loki must keep, which corrupts the store and causes query failures. The note at the top of this page applies here as well: any lifecycle expiration must be longer than your retention period.\n\n> Warning\n> \n> Never apply a lifecycle rule with an empty prefix (one that targets the whole bucket). Always scope lifecycle rules to the per-tenant chunk prefixes described below.\n\n### Bucket object layout \n\nThe following objects are written to the configured bucket. The exact prefixes depend on your `schema_config` and on whether multi-tenancy is enabled, so confirm them against your own configuration before writing a lifecycle rule.\n\n| Object or prefix | Content | Safe to expire with a lifecycle policy? |\n| --- | --- | --- |\n| `/` (for example, `fake/` in single-tenant mode, or your org ID per tenant) | Chunk objects (the log data). | Yes. These are the only objects a lifecycle rule should target. |\n| Index objects (under the index path prefix, `index/` by default, set by `index.path_prefix` in `schema_config`) | TSDB or BoltDB index files, written by ingesters and rewritten by the Compactor. The `index.prefix` value (for example `index_` or `loki_index_`) is the table-name prefix used inside this path, not the path itself, so match a lifecycle rule against the path prefix. | No. Loki manages the index lifecycle internally. Deleting index objects removes references to live chunks and breaks queries. |\n| `loki_cluster_seed.json` | A small file at the bucket root holding the cluster seed used by usage reporting. It is written once and never rotated. | No. A blanket age-based rule will delete it once it ages past the time to live (TTL). |\n| Compactor delete-request store (the `delete_request_store` you configure, such as `markers/` and the delete-request objects) | The Compactor\u2019s record of chunks pending deletion and of in-flight delete requests. | No. Removing these makes the Compactor lose track of pending deletions. |\n| Ruler store objects (if the Ruler is backed by the same bucket) | Recording and alerting rule files. | No. These are configuration, not time-series data. |\n\n### Compactor retention versus object store lifecycle policies \n\nCompactor retention and object store lifecycle policies are two independent mechanisms, and they should not be configured in ways that conflict:\n\n- The Compactor is index-aware. It only deletes a chunk after it has removed every reference to that chunk from the index, and it waits `compactor.retention-delete-delay` (default `2h`) before the sweeper deletes the marked chunk, so that index gateways pick up the rewritten index first.\n- An object store lifecycle policy is index-unaware. It deletes objects purely by age and prefix, with no knowledge of whether a chunk is still referenced by the index.\n\nFor most deployments, enabling Compactor retention is sufficient and an object store lifecycle policy is not required. If you do add a lifecycle policy as a cost safety net, set its expiration longer than your retention period plus `compactor.retention-delete-delay`, so the Compactor always deletes chunks first and the lifecycle rule only ever catches objects the Compactor has already orphaned.\n\n### Example: Amazon S3 lifecycle policy \n\nThis policy expires only chunk objects under a single tenant prefix. Replace `fake/` with your tenant prefix (use one rule per tenant, or your org ID prefix, when multi-tenancy is enabled). The `Days` value must be larger than your retention period plus the delete delay.\n\n```json\n{\n \"Rules\": [\n {\n \"ID\": \"loki-chunk-expiration\",\n \"Status\": \"Enabled\",\n \"Filter\": {\n \"Prefix\": \"fake/\"\n },\n \"Expiration\": {\n \"Days\": 395\n }\n }\n ]\n}\n```\n\nApply it with the AWS CLI:\n\n```bash\naws s3api put-bucket-lifecycle-configuration \\\n --bucket YOUR_LOKI_BUCKET \\\n --lifecycle-configuration file://lifecycle.json\n```\n\n### Example: Google Cloud Storage lifecycle rule \n\nThe equivalent GCS rule expires objects by age and prefix. As with S3, scope `matchesPrefix` to the tenant chunk prefix and keep `age` larger than your retention period plus the delete delay.\n\n```json\n{\n \"rule\": [\n {\n \"action\": {\"type\": \"Delete\"},\n \"condition\": {\n \"age\": 395,\n \"matchesPrefix\": [\"fake/\"]\n }\n }\n ]\n}\n```\n\nApply it with the Google Cloud CLI:\n\n```bash\ngcloud storage buckets update gs://YOUR_LOKI_BUCKET --lifecycle-file=lifecycle.json\n```\n\nFor Azure Blob Storage, use a management policy with a `prefixMatch` set to the tenant chunk prefix and the same \u201cexpiration must exceed retention plus delete delay\u201d rule.\n\nScaling and securing your logs with Grafana Loki\n\nThis webinar covers the challenges of scaling and securing logs, and how Grafana Cloud Logs powered by Grafana Loki can help, cost-effectively.\n\nOn this page\n\n See what\u2019s actually working with AI in observability. Learn how teams are using AI for investigations, incident response, OpenTelemetry, and telemetry cost control in production. \n\n Bring your hardest observability problem. From OpenTelemetry and incident response to telemetry cost control and AI-powered investigations, learn what\u2019s working at scale. \n\n Learn how teams are scaling observability with AI, OpenTelemetry, and Adaptive Telemetry. Real systems. Real incidents. Real operational lessons.", "content_chars": 16394, "published_date": null }, { "rank": 4, "url": "https://github.com/grafana/loki/issues/6300", "title": "Old chunks not getting deleted after retention period", "content": "# Old chunks not getting deleted after retention period\n\n- State: closed\n- Author: wzjjack\n- Created: 2022-06-03T06:31:56Z\n- Updated: 2026-04-22T00:46:55Z\n- Repository: grafana/loki\n- Number: #6300\n\n---\n\n \n \n**Describe the bug** \nI've configured 168h retention for my logs, but I can see chunks 5 years old filling my disk \n \n**To Reproduce** \n \nthis is my config \n```\r\nauth_enabled: false\r\n\r\nserver:\r\n http_listen_port: 3100\r\n grpc_listen_port: 9096\r\n grpc_server_max_recv_msg_size: 8388608\r\n grpc_server_max_send_msg_size: 8388608\r\nquerier:\r\n engine:\r\n max_look_back_period: 168h \r\n\r\ningester:\r\n wal:\r\n enabled: true\r\n dir: /tmp/wal\r\n lifecycler:\r\n address: 127.0.0.1\r\n ring:\r\n kvstore:\r\n store: inmemory\r\n replication_factor: 1\r\n final_sleep: 0s\r\n chunk_idle_period: 24h # Any chunk not receiving new logs in this time will be flushed\r\n max_chunk_age: 24h # All chunks will be flushed when they hit this age, default is 1h\r\n chunk_target_size: 1048576 # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first\r\n chunk_retain_period: 5m # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)\r\n max_transfer_retries: 0 # Chunk transfers disabled\r\n\r\nschema_config:\r\n configs:\r\n - from: 2020-10-24\r\n store: boltdb-shipper\r\n object_store: filesystem\r\n schema: v11\r\n index:\r\n prefix: index_\r\n period: 24h\r\n\r\nstorage_config:\r\n boltdb_shipper:\r\n active_index_directory: /tmp/loki/boltdb-shipper-active\r\n cache_location: /tmp/loki/boltdb-shipper-cache\r\n cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space\r\n shared_store: filesystem\r\n filesystem:\r\n directory: /tmp/loki/chunks\r\n\r\ncompactor:\r\n working_directory: /tmp/loki/boltdb-shipper-compactor\r\n shared_store: filesystem\r\n\r\nlimits_config:\r\n reject_old_samples: true\r\n reject_old_samples_max_age: 168h\r\n max_streams_per_user: 1000000\r\n max_entries_limit_per_query: 5000000\r\n ingestion_rate_mb: 100\r\n ingestion_burst_size_mb: 20 \r\nchunk_store_config:\r\n max_look_back_period: 168h\r\n\r\ntable_manager:\r\n retention_deletes_enabled: false\r\n retention_period: 168h\r\n```\r\n \n**Expected behavior** \nChunks older than 168h should be deleted. \n \n**Environment:** \n - Infrastructure: [e.g., Kubernetes, bare-metal, laptop] \n - Deployment tool: [e.g., helm, jsonnet] \n \n**Screenshots, Promtail config, or terminal output** \nWe can see 49 days of logs although I've configured 168h \nimage \n\n## Timeline\n\n- Referenced by issue #936: Old chunks not getting deleted after retention period\n\n**DeBuXer** commented on 2022-06-03T13:04:57Z:\n\n> I got the same issue. Logs older than 7 days are deleted and they're not visible in Grafana. Only the chunk files won't be deleted on the filesystem.\n> \n> ```\n> loki, version 2.5.0 (branch: HEAD, revision: 2d9d0ee23)\n> build user: root@4779f4b48f3a\n> build date: 2022-04-07T21:50:00Z\n> go version: go1.17.6\n> platform: linux/amd64\n> ```\n> \n> ```\n> server:\n> http_listen_port: 3100\n> grpc_listen_port: 9096\n> \n> common:\n> path_prefix: /var/lib/loki\n> storage:\n> filesystem:\n> chunks_directory: /var/lib/loki/chunks\n> rules_directory: /var/lib/loki/rules\n> replication_factor: 1\n> ring:\n> instance_addr: 127.0.0.1\n> kvstore:\n> store: inmemory\n> \n> schema_config:\n> configs:\n> - from: 2020-10-24\n> store: boltdb-shipper\n> object_store: filesystem\n> schema: v11\n> index:\n> prefix: index_\n> period: 24h\n> \n> chunk_store_config:\n> max_look_back_period: 168h\n> \n> table_manager:\n> retention_deletes_enabled: true\n> retention_period: 168h\n> ```\n\n**splitice** commented on 2022-06-04T07:15:14Z:\n\n> Use compactor not table_manager if you arent using AWS S3.\n\n**DeBuXer** commented on 2022-06-07T08:52:38Z:\n\n> > Use compactor not table_manager if you arent using AWS S3.\n> \n> Thanks, that did the trick :)\n\n**Mastedont** commented on 2022-06-14T09:56:43Z:\n\n> Hey @DeBuXer, could you post what you add to your config file in order to get deletion on s3 done?. I am running the same issue and I have not found the solution.\n\n- DeBuXer mentioned\n\n- DeBuXer subscribed\n\n**DeBuXer** commented on 2022-06-14T10:00:18Z:\n\n> @Mastedont, I don't use S3, I store my chunks directly on disk. My current configuration:\n> \n> ```\n> server:\n> http_listen_port: 3100\n> grpc_listen_port: 9096\n> \n> common:\n> path_prefix: /var/lib/loki\n> storage:\n> filesystem:\n> chunks_directory: /var/lib/loki/chunks\n> rules_directory: /var/lib/loki/rules\n> replication_factor: 1\n> ring:\n> instance_addr: 127.0.0.1\n> kvstore:\n> store: inmemory\n> \n> schema_config:\n> configs:\n> - from: 2020-10-24\n> store: boltdb-shipper\n> object_store: filesystem\n> schema: v11\n> index:\n> prefix: index_\n> period: 24h\n> \n> chunk_store_config:\n> max_look_back_period: 168h\n> \n> compactor:\n> working_directory: /var/lib/loki/retention\n> shared_store: filesystem\n> compaction_interval: 10m\n> retention_enabled: true\n> retention_delete_delay: 2h\n> retention_delete_worker_count: 150\n> \n> limits_config:\n> retention_period: 168h\n> \n> ruler:\n> alertmanager_url: http://127.0.0.1:9093\n> ```\n\n- Mastedont mentioned\n\n- Mastedont subscribed\n\n**Mastedont** commented on 2022-06-14T10:02:00Z:\n\n> Thank you, @DeBuXer \n\n- DeBuXer mentioned\n\n- DeBuXer subscribed\n\n**Mastedont** commented on 2022-06-14T11:02:51Z:\n\n> One last question @DeBuXer.\n> \n> Do you how can I know if log retention is working or not? What output written in logs will let know thats is working?\n\n- DeBuXer mentioned\n\n- DeBuXer subscribed\n\n**DeBuXer** commented on 2022-06-14T13:19:51Z:\n\n> @Mastedont, Not 100% sure, but I guess:\n> \n> ```\n> Jun 14 15:15:44 loki loki[277929]: level=info ts=2022-06-14T13:15:44.57537489Z caller=index_set.go:280 table-name=index_19150 msg=\"removing source db files from storage\" count=1\n> Jun 14 15:15:44 loki loki[277929]: level=info ts=2022-06-14T13:15:44.576099223Z caller=compactor.go:495 msg=\"finished compacting table\" table-name=index_19150\n> ```\n\n- Mastedont mentioned\n\n- Mastedont subscribed\n\n**Mastedont** commented on 2022-06-14T14:07:09Z:\n\n> That log output is from Ingester?\n> \n> I only can see outpur like this, despite of having Compactor enabled:\n> \n> ```\n> level=info ts=2022-06-14T14:05:17.574831148Z caller=table.go:358 msg=\"uploading table loki_pre_19157\"\n> level=info ts=2022-06-14T14:05:17.574847901Z caller=table.go:385 msg=\"finished uploading table loki_pre_19157\"\n> level=info ts=2022-06-14T14:05:17.57485537Z caller=table.go:443 msg=\"cleaning up unwanted dbs from table loki_pre_19157\"\n> ```\n\n**DeBuXer** commented on 2022-06-14T14:26:06Z:\n\n> > That log output is from Ingester?\n> \n> From /var/log/syslog but should have the same information. When compactor is enabled, you should see something like;\n> \n> ```\n> level=info ts=2022-06-14T14:24:56.072803949Z caller=compactor.go:324 msg=\"this instance has been chosen to run the compactor, starting compactor\"\n> ```\n\n**rickydjohn** commented on 2022-07-05T07:19:52Z:\n\n> @DeBuXer , thanks a lot for your support here. I don't see the chunk files getting rotated. I also see pretty old index directories as well. I wanted my logs to be rotated every 7 days. I am not sure what I am doing wrong here. Could you please help me with it?\n> \n> ```\n> auth_enabled: false\n> chunk_store_config:\n> max_look_back_period: 168h\n> \n> compactor:\n> shared_store: filesystem\n> working_directory: /data/loki/boltdb-shipper-compactor\n> \n> ingester:\n> chunk_block_size: 262144\n> chunk_idle_period: 3m\n> chunk_retain_period: 1m\n> wal:\n> dir: /data/loki/wal \n> flush_on_shutdown: true\n> lifecycler:\n> ring:\n> kvstore:\n> store: inmemory\n> replication_factor: 1\n> max_transfer_retries: 0\n> \n> limits_config:\n> enforce_metric_name: false\n> reject_old_samples: true\n> reject_old_samples_max_age: 168h\n> ingestion_rate_mb: 32\n> ingestion_burst_size_mb: 36\n> unordered_writes: true\n> retention_period: 168h\n> \n> schema_config:\n> configs:\n> - from: 2020-10-24\n> index:\n> period: 24h\n> prefix: index_\n> object_store: filesystem\n> schema: v11\n> store: boltdb-shipper\n> \n> server:\n> http_listen_port: 3100\n> \n> storage_config:\n> boltdb_shipper:\n> active_index_directory: /data/loki/boltdb-shipper-active\n> cache_location: /data/loki/boltdb-shipper-cache\n> cache_ttl: 24h\n> shared_store: filesystem\n> filesystem:\n> directory: /data/loki/chunks\n> \n> table_manager:\n> retention_deletes_enabled: true\n> retention_period: 168h\n> \n> ```\n\n- DeBuXer mentioned\n\n- DeBuXer subscribed\n\n**DeBuXer** commented on 2022-07-05T07:30:31Z:\n\n> @rickydjohn, I think you need to enable ` retention_enabled`. See also https://grafana.com/docs/loki/latest/operations/storage/retention/#retention-configuration\n\n- rickydjohn mentioned\n\n- rickydjohn subscribed\n\n**stale[bot]** commented on 2022-08-13T12:49:02Z:\n\n> Hi! This issue has been automatically marked as stale because it has not had any\n> activity in the past 30 days.\n> \n> We use a stalebot among other tools to help manage the state of issues in this project.\n> A stalebot can be very useful in closing issues in a number of cases; the most common\n> is closing issues or PRs where the original reporter has not responded.\n> \n> Stalebots are also emotionless and cruel and can close issues which are still very relevant.\n> \n> **If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.**\n> \n> We regularly sort for closed issues which have a `stale` label sorted by thumbs up.\n> \n> We may also:\n> * Mark issues as `revivable` if we think it's a valid issue but isn't something we are likely\n> to prioritize in the future (the issue will still remain closed).\n> * Add a `keepalive` label to silence the stalebot if the issue is very common/popular/important.\n> \n> We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,\n> our sincere apologies if you find yourself at the mercy of the stalebot.\n\n- stale[bot] added label \"stale\"\n\n**atze234** commented on 2022-08-30T08:20:16Z:\n\n> Hi @Mastedont ,\n> did you manage to get the chunks deleted from s3? Im having that same problem, that i cannot see any logs about compactor and in s3 store there are older files (>7days) than my configured retention. It seems only the index is cleared, because grafana wont show older log entries. \n\n- Mastedont mentioned\n\n- Mastedont subscribed\n\n**ghost** commented on 2022-09-05T07:10:20Z:\n\n> > Hi @Mastedont , did you manage to get the chunks deleted from s3? Im having that same problem, that i cannot see any logs about compactor and in s3 store there are older files (>7days) than my configured retention. It seems only the index is cleared, because grafana wont show older log entries.\n> \n> I have the same problem\n\n- Mastedont mentioned\n\n- Mastedont subscribed\n\n- stale[bot] removed label \"stale\"\n\n**webfrank** commented on 2022-11-16T17:11:14Z:\n\n> Hi, I have this relevant config:\n> \n> ```\n> compactor:\n> compaction_interval: 10m\n> retention_delete_delay: 2h\n> retention_delete_worker_count: 150\n> retention_enabled: true\n> shared_store: s3\n> working_directory: /var/loki/retention\n> limits_config:\n> enforce_metric_name: false\n> max_cache_freshness_per_query: 10m\n> reject_old_samples: true\n> reject_old_samples_max_age: 168h\n> retention_period: 720h\n> split_queries_by_interval: 30m\n> ```\n> \n> but log files are not deleted on S3, only index compacted.\n\n**jarrettprosser** commented on 2022-11-18T03:32:44Z:\n\n> I'm also finding this on loki 2.4.0, using minio as storage. Even with `retention_delete_delay: 5m` no chunks are being deleted. \n\n**Codecaver** commented on 2022-12-19T08:13:04Z:\n\n> > @Mastedont, I don't use S3, I store my chunks directly on disk. My current configuration:\n> > \n> > ```\n> > server:\n> > http_listen_port: 3100\n> > grpc_listen_port: 9096\n> > \n> > common:\n> > path_prefix: /var/lib/loki\n> > storage:\n> > filesystem:\n> > chunks_directory: /var/lib/loki/chunks\n> > rules_directory: /var/lib/loki/rules\n> > replication_factor: 1\n> > ring:\n> > instance_addr: 127.0.0.1\n> > kvstore:\n> > store: inmemory\n> > \n> > schema_config:\n> > configs:\n> > - from: 2020-10-24\n> > store: boltdb-shipper\n> > object_store: filesystem\n> > schema: v11\n> > index:\n> > prefix: index_\n> > period: 24h\n> > \n> > chunk_store_config:\n> > max_look_back_period: 168h\n> > \n> > compactor:\n> > working_directory: /var/lib/loki/retention\n> > shared_store: filesystem\n> > compaction_interval: 10m\n> > retention_enabled: true\n> > retention_delete_delay: 2h\n> > retention_delete_worker_count: 150\n> > \n> > limits_config:\n> > retention_period: 168h\n> > \n> > ruler:\n> > alertmanager_url: http://127.0.0.1:9093\n> > ```\n> Hi\uff0cwill this configuration clean up expired files in the chunks directory?\n\n- Mastedont mentioned\n\n- Mastedont subscribed\n\n**patsevanton** commented on 2022-12-30T10:38:28Z:\n\n> any update ?\n\n**seany89** commented on 2023-01-27T13:29:47Z:\n\n> Judging from the discussion in this issue https://github.com/grafana/loki/issues/7068\n> I don't think the compactor will delete the chunks in s3 object store you need a bucket lifecycle policy for that.\n> \n> It would be nice to have a clear answer on this though.\n\n**nvanheuverzwijn** commented on 2023-02-15T19:16:38Z:\n\n> For everyone wondering what's going with the retention, I've tested the feature _a lot_ in the past days so here are what will work.\n> \n> ### Minimal Configuration Needed\n> First of all, you absolutely need those config setup\n> ```\n> limits_config:\n> retention_period: 10d # Keep 10 days\n> compactor:\n> delete_request_cancel_period: 10m # don't wait 24h before processing the delete_request\n> retention_enabled: true # actually do the delete\n> retention_delete_delay: 2h # wait 2 hours before actually deleting stuff\n> ```\n> You can tweak those config to delete faster or slower.\n> \n> ### Check If It's Working\n> Once you got those config up and running, check that the logs are actually reporting that the retention is being applied : `msg=\"applying retention with compaction\"`. The \"caller\" for this log is `compactor.go`.\n> \n> Next, check that the retention manager is actually doing it's job in the logs: `msg=\"mark file created\"` and `msg=\"no marks file found\"` from the caller `marker.go`.\n> \n> The `mark file created` means that loki did found some chunks to be deleted and it has created a file to keep track of it. The `no marks file found` means that while performing the chunk delete routine, there was no file that matched it's filters, the filters mainly being the delay.\n> \n> Whenever you see the `mark file created` logs, you can go into the working directory of the compactor and check for the mark files. The path should be something like `/var/loki/compactor/retention/markers`. These files are kept there for 2 hours or whatever is set in `retention_delete_delay`. After `retention_delete_delay` is passed, loki _will_ delete the chunks.\n> \n> Not having any of the logs mentionned above means that the retention process is not started.\n> \n> ### Important Notes\n> Loki will *only delete chunks that are indexed*. The indexes are actually being purged before deleting the chunks. This means that if you lose files from the compactor's working directory, whatever chunks that were marked there won't be deleted ever so it is still worth to have a lifecycle policy to cover for this OR have persistent storage for this particular folder.\n\n**amseager** commented on 2023-03-05T13:27:48Z:\n\n> @nvanheuverzwijn if I were the CTO of Grafana Labs, I would give you a job offer immediately\n\n- nvanheuverzwijn mentioned\n\n- nvanheuverzwijn subscribed\n\n**adthonb** commented on 2023-04-28T00:54:14Z:\n\n> @nvanheuverzwijn Thank you a lot! Your explanation makes me clear. The Loki document makes me confuse that Table Manager also deletes chucks when using the filesystem chuck store.\n\n- nvanheuverzwijn mentioned\n\n- nvanheuverzwijn subscribed\n\n**yangmeilly** commented on 2023-07-11T12:48:42Z:\n\n> more info about 'Check If It's Working'\n> such as `compaction_interval: 10m`\n> assuming the loki instance start at `2023-07-11T12:30:25.060395045Z`, then there are logs about caller=compactor.go at `ts=2023-07-11T12:40:25.047110295Z`\n> \n> level=info ts=2023-07-11T12:30:25.060441045Z caller=compactor.go:440 msg=\"waiting 10m0s for ring to stay stable and previous compactions to finish before starting compactor\"\n> level=info ts=2023-07-11T12:40:25.045542628Z caller=compactor.go:445 msg=\"compactor startup delay completed\"\n> level=info ts=2023-07-11T12:40:25.045568295Z caller=compactor.go:497 msg=\"compactor started\"\n> **level=info ts=2023-07-11T12:40:25.04562367Z caller=compactor.go:454 msg=\"applying retention with compaction\"**\n> level=info ts=2023-07-11T12:40:25.047110295Z caller=compactor.go:609 msg=\"compacting table\" table-name=loki_index_19549\n> level=info ts=2023-07-11T12:40:25.047208753Z caller=table_compactor.go:325 table-name=loki_index_19549 msg=\"using compactor-1689078092.gz as seed file\"\n> level=info ts=2023-07-11T12:40:25.048495753Z caller=util.go:85 table-name=loki_index_19549 file-name=compactor-1689078092.gz msg=\"downloaded file\" total_time=1.280041ms\n> level=info ts=2023-07-11T12:40:25.06665592Z caller=compactor.go:614 msg=\"finished compacting table\" table-name=loki_index_19549\n> level=info ts=2023-07-11T12:40:25.066668503Z caller=compactor.go:609 msg=\"compacting table\" table-name=loki_index_19548\n> level=info ts=2023-07-11T12:40:25.067591628Z caller=util.go:85 table-name=loki_index_19548 file-name=compactor-1689041382.gz msg=\"downloaded file\" total_time=863.125\u00b5s\n> level=info ts=2023-07-11T12:40:25.078401878Z caller=compactor.go:614 msg=\"finished compacting table\" table-name=loki_index_19548\n\n**Nurlan199206** commented on 2023-07-12T07:56:53Z:\n\n> @yangmeilly \n> \n> you can send full config pls `loki.yaml`?\n> \n> for me not working.\n\n- yangmeilly mentioned\n\n- yangmeilly subscribed\n\n**yangmeilly** commented on 2023-07-13T08:30:15Z:\n\n> > @yangmeilly\n> > \n> > you can send full config pls `loki.yaml`?\n> > \n> > for me not working.\n> \n> in my scenario, using boltdb-shipper for indexs and filesystem for chunks.\n> the full loki config as following, and bold should have your attention.\n> \n> \n> **compactor block:\n> compaction_interval: 10m\n> delete_request_cancel_period: 2h\n> retention_delete_delay: 2h\n> retention_delete_worker_count: 150\n> retention_enabled: true\n> shared_store: filesystem\n> working_directory: /var/loki/retention**\n> \n> limits_config block:\n> enforce_metric_name: false\n> max_cache_freshness_per_query: 10m\n> reject_old_samples: true\n> reject_old_samples_max_age: 168h\n> split_queries_by_interval: 15m\n> **etention_period: 72h\n> max_query_lookback: 72h**\n> \n> \n> table_manager block: // this make nosense for filesystem\n> retention_deletes_enabled: false\n> retention_period: 0\n\n- yangmeilly mentioned\n\n- yangmeilly subscribed\n\n**HammerNL89** commented on 2023-07-25T12:55:21Z:\n\n> > Whenever you see the `mark file created` logs, you can go into the working directory of the compactor and check for the mark files. The path should be something like `/var/loki/compactor/retention/markers`. These files are kept there for 2 hours or whatever is set in `retention_delete_delay`. After `retention_delete_delay` is passed, loki _will_ delete the chunks.\n> > \n> > Not having any of the logs mentionned above means that the retention process is not started.\n> > \n> \n> @nvanheuverzwijn\n> Thanks for the info. Regarding your statement, loki will delete the chunks, are you talking about a filesystem backend or also a s3/azure backend? \n> I can't find a definitive answer stating that loki is able to delete chunks from external storage.\n\n- nvanheuverzwijn mentioned\n\n- nvanheuverzwijn subscribed\n\n**nvanheuverzwijn** commented on 2023-07-25T13:09:29Z:\n\n> It will also delete on S3/Azure. I did this with Google Cloud storage but\n> it should be the same for the other backend.\n> \n> Le mar. 25 juill. 2023, 08 h 55, HammerNL89 ***@***.***> a\n> \u00e9crit :\n> \n> > Whenever you see the mark file created logs, you can go into the working\n> > directory of the compactor and check for the mark files. The path should be\n> > something like /var/loki/compactor/retention/markers. These files are\n> > kept there for 2 hours or whatever is set in retention_delete_delay.\n> > After retention_delete_delay is passed, loki *will* delete the chunks.\n> >\n> > Not having any of the logs mentionned above means that the retention\n> > process is not started.\n> >\n> > @nvanheuverzwijn \n> > Thanks for the info. Regarding your statement, loki will delete the\n> > chunks, are you talking about a filesystem backend or also a s3/azure\n> > backend?\n> > I can't find a definitive answer stating that loki is able to delete\n> > chunks from external storage.\n> >\n> > \u2014\n> > Reply to this email directly, view it on GitHub\n> >, or\n> > unsubscribe\n> > \n> > .\n> > You are receiving this because you were mentioned.Message ID:\n> > ***@***.***>\n> >\n\n- nvanheuverzwijn mentioned\n\n- nvanheuverzwijn subscribed\n\n**stringang** commented on 2023-08-03T10:59:23Z:\n\n> @nvanheuverzwijn \n> `compactor` did not delete the chunk. why?\n> \n> compactor log:\n> ```\n> level=info ts=2023-08-03T06:50:12.634846248Z caller=compactor.go:497 msg=\"compactor started\"\n> level=info ts=2023-08-03T06:50:12.634865722Z caller=compactor.go:454 msg=\"applying retention with compaction\"\n> level=info ts=2023-08-03T06:50:12.634865349Z caller=marker.go:177 msg=\"mark processor started\" workers=150 delay=2h0m0s\n> level=info ts=2023-08-03T06:50:12.634955656Z caller=expiration.go:78 msg=\"overall smallest retention period 1690440612.634, default smallest retention period 1690440612.634\"\n> ts=2023-08-03T06:50:12.635021334Z caller=spanlogger.go:85 level=info msg=\"building index list cache\"\n> level=info ts=2023-08-03T06:50:12.635046761Z caller=marker.go:202 msg=\"no marks file found\"\n> ```\n> \n> config:\n> ```\n> storage_config:\n> aws:\n> access_key_id: xxxxxx\n> bucketnames: loki\n> endpoint: https://s3.xxxx.com\n> s3forcepathstyle: true\n> secret_access_key: xxxxx\n> boltdb_shipper:\n> active_index_directory: /var/loki/index\n> cache_location: /var/loki/cache\n> cache_ttl: 24h\n> index_gateway_client:\n> server_address: dns:///loki-distributed-index-gateway:9095\n> shared_store: s3\n> \n> compactor:\n> retention_enabled: true\n> shared_store: s3\n> working_directory: /var/loki/compactor\n> retention_delete_delay: 2h\n> delete_request_cancel_period: 10m\n> \n> limits_config:\n> enforce_metric_name: false\n> ingestion_burst_size_mb: 1024\n> ingestion_rate_mb: 1024\n> max_cache_freshness_per_query: 10m\n> max_global_streams_per_user: 0\n> reject_old_samples: true\n> reject_old_samples_max_age: 168h\n> retention_period: 1h\n> split_queries_by_interval: 15m\n> ```\n> ----\n> **update**:\n> It was caused by my incorrect configuration.\n> \n> The storage configuration needs to be placed in the `common`.\n> \n> ```\n> common:\n> compactor_address: http://loki-distributed-compactor:3100\n> storage:\n> s3:\n> access_key_id: xxxxxx\n> bucketnames: loki\n> endpoint: https://s3.xxxx.com\n> s3forcepathstyle: true\n> secret_access_key: xxxxxx\n> ```\n> \n> \n\n- nvanheuverzwijn mentioned\n\n- nvanheuverzwijn subscribed\n\n**yangfan-witcher** commented on 2023-08-15T09:20:10Z:\n\n> @nvanheuverzwijn so beautiful\n\n- nvanheuverzwijn mentioned\n\n- nvanheuverzwijn subscribed\n\n**ningyougang** commented on 2023-09-05T05:41:47Z:\n\n> @stringang can you share the whole loki.yaml?\n> \n> I am also testing, log files are not deleted from S3, only index compacted,\n> 1. index compacted, e.g.\n> image\n> 2. log files are not deleted, e.g.\n> image\n> \n> My whole configuartion\n> ```\n> auth_enabled: false\n> chunk_store_config:\n> max_look_back_period: 0s\n> compactor:\n> shared_store: s3\n> working_directory: /data/loki/boltdb-shipper-compactor\n> retention_enabled: true\n> compaction_interval: 10m\n> retention_delete_delay: 2h\n> distributor:\n> ring:\n> kvstore:\n> store: memberlist\n> ingester:\n> chunk_block_size: 262144\n> chunk_idle_period: 3m\n> chunk_retain_period: 1m\n> lifecycler:\n> ring:\n> kvstore:\n> store: memberlist\n> replication_factor: 1\n> max_transfer_retries: 0\n> query_store_max_look_back_period: 0\n> wal:\n> enabled: true\n> dir: /data/wal\n> querier:\n> max_concurrent: 20\n> limits_config:\n> ingestion_rate_mb: 8\n> ingestion_burst_size_mb: 16\n> per_stream_rate_limit: 5MB\n> per_stream_rate_limit_burst: 15MB\n> enforce_metric_name: false\n> reject_old_samples: true\n> reject_old_samples_max_age: 24h\n> retention_period: 24h\n> memberlist:\n> join_members:\n> - loki-headless\n> schema_config:\n> configs:\n> - from: \"2022-11-20\"\n> store: boltdb-shipper\n> object_store: s3\n> schema: v11\n> index:\n> period: 24h\n> prefix: index_\n> chunks:\n> period: 24h\n> prefix: chunks_\n> server:\n> http_listen_port: 3100\n> log_level: debug\n> storage_config:\n> object_prefix: test-nyg\n> boltdb_shipper:\n> active_index_directory: /data/loki/boltdb-shipper-active\n> cache_location: /data/loki/boltdb-shipper-cache\n> cache_ttl: 24h\n> shared_store: s3\n> common:\n> storage:\n> s3:\n> s3: s3://test-admin:test-admin@s3Address:10000/$bucketName\n> s3forcepathstyle: true\n> ```\n> \n> Do i miss something?\n\n- stringang mentioned\n\n- stringang subscribed\n\n**stringang** commented on 2023-09-05T07:02:40Z:\n\n> @ningyougang \n> ```\n> auth_enabled: false\n> chunk_store_config:\n> max_look_back_period: 0s\n> common:\n> compactor_address: http://loki-distributed-compactor:3100\n> storage:\n> s3:\n> access_key_id: xxxxxxxxxxx\n> bucketnames: loki\n> endpoint: https://s3.xxxxxx.com\n> s3forcepathstyle: true\n> secret_access_key: xxxxxxxxxxx\n> compactor:\n> delete_request_cancel_period: 10m\n> retention_delete_delay: 1h\n> retention_enabled: true\n> shared_store: s3\n> working_directory: /var/loki/compactor\n> distributor:\n> ring:\n> kvstore:\n> store: memberlist\n> frontend:\n> compress_responses: true\n> log_queries_longer_than: 5s\n> tail_proxy_url: http://loki-distributed-querier:3100\n> frontend_worker:\n> frontend_address: loki-distributed-query-frontend-headless:9095\n> ingester:\n> chunk_block_size: 262144\n> chunk_encoding: snappy\n> chunk_idle_period: 1h\n> chunk_retain_period: 1m\n> chunk_target_size: 8388608\n> lifecycler:\n> join_after: 10s\n> observe_period: 5s\n> ring:\n> heartbeat_timeout: 10m\n> kvstore:\n> store: memberlist\n> replication_factor: 3\n> max_transfer_retries: 0\n> wal:\n> dir: /var/loki/wal\n> ingester_client:\n> grpc_client_config:\n> grpc_compression: gzip\n> limits_config:\n> enforce_metric_name: false\n> ingestion_burst_size_mb: 1024\n> ingestion_rate_mb: 1024\n> max_cache_freshness_per_query: 10m\n> max_global_streams_per_user: 0\n> reject_old_samples: true\n> reject_old_samples_max_age: 168h\n> retention_period: 1d\n> split_queries_by_interval: 15m\n> memberlist:\n> join_members:\n> - loki-distributed-memberlist\n> query_range:\n> align_queries_with_step: true\n> cache_results: true\n> max_retries: 5\n> results_cache:\n> cache:\n> embedded_cache:\n> enabled: true\n> ttl: 24h\n> ruler:\n> alertmanager_url: http://am.xxxxxxxxxxx.com/\n> enable_alertmanager_v2: true\n> enable_api: true\n> enable_sharding: true\n> ring:\n> kvstore:\n> store: memberlist\n> rule_path: /tmp/loki/scratch\n> storage:\n> local:\n> directory: /etc/loki/rules\n> type: local\n> runtime_config:\n> file: /var/loki-distributed-runtime/runtime.yaml\n> schema_config:\n> configs:\n> - from: \"2023-08-12\"\n> index:\n> period: 24h\n> prefix: loki_index_\n> object_store: s3\n> schema: v11\n> store: boltdb-shipper\n> server:\n> grpc_server_max_recv_msg_size: 8388608\n> http_listen_port: 3100\n> log_level: debug\n> storage_config:\n> boltdb_shipper:\n> active_index_directory: /var/loki/index\n> cache_location: /var/loki/cache\n> cache_ttl: 24h\n> index_gateway_client:\n> server_address: dns:///loki-distributed-index-gateway:9095\n> shared_store: s3\n> table_manager:\n> retention_deletes_enabled: false\n> retention_period: 0s\n> ```\n\n- ningyougang mentioned\n\n- ningyougang subscribed\n\n**ben5556** commented on 2023-09-11T05:29:15Z:\n\n> @nvanheuverzwijn I do see mark file created but then a few mins later the logs show\n> ```\n> level=info ts=2023-09-11T05:23:41.019592474Z caller=compactor.go:364 msg=\"applying retention with compaction\"\n> level=info ts=2023-09-11T05:23:41.020937376Z caller=expiration.go:60 msg=\"overall smallest retention period 1691817821.02, default smallest retention period 1691817821.02\"\n> level=info ts=2023-09-11T05:23:41.065287615Z caller=marker.go:78 msg=\"mark file created\" file=/data/loki/boltdb-shipper-compactor/retention/markers/1694409821056046054\n> ```\n> \n> Few secs later\n> ```\n> level=info ts=2023-09-11T05:24:41.019702032Z caller=marker.go:203 msg=\"no marks file found\"\n> level=info ts=2023-09-11T05:25:41.020103387Z caller=marker.go:203 msg=\"no marks file found\"\n> level=info ts=2023-09-11T05:26:41.020436386Z caller=marker.go:203 msg=\"no marks file found\"\n> level=info ts=2023-09-11T05:27:41.020020721Z caller=marker.go:203 msg=\"no marks file found\"\n> level=info ts=2023-09-11T05:28:41.01973026Z caller=marker.go:203 msg=\"no marks file found\"\n> level=info ts=2023-09-11T05:30:41.020528317Z caller=marker.go:203 msg=\"no marks file found\"\n> ```\n> \n> So looks like even after 5 mins (retention_delete_delay) it still cannot find the mark file. I verified that the mark file exists in that location\n> Below is my loki config\n> \n> ```\n> auth_enabled: false\n> chunk_store_config:\n> max_look_back_period: 0s\n> compactor:\n> delete_request_cancel_period: 10m\n> retention_delete_delay: 5m\n> retention_delete_worker_count: 150\n> retention_enabled: true\n> shared_store: filesystem\n> working_directory: /data/loki/boltdb-shipper-compactor\n> ingester:\n> chunk_block_size: 262144\n> chunk_idle_period: 3m\n> chunk_retain_period: 1m\n> lifecycler:\n> ring:\n> replication_factor: 1\n> max_transfer_retries: 0\n> wal:\n> dir: /data/loki/wal\n> limits_config:\n> enforce_metric_name: false\n> max_entries_limit_per_query: 5000\n> max_query_lookback: 720h\n> reject_old_samples: true\n> reject_old_samples_max_age: 168h\n> retention_period: 720h\n> memberlist:\n> join_members:\n> - 'loki-memberlist'\n> schema_config:\n> configs:\n> - from: \"2020-10-24\"\n> index:\n> period: 24h\n> prefix: index_\n> object_store: filesystem\n> schema: v11\n> store: boltdb-shipper\n> server:\n> grpc_listen_port: 9095\n> http_listen_port: 3100\n> http_server_read_timeout: 120s\n> ```\n\n- nvanheuverzwijn mentioned\n\n- nvanheuverzwijn subscribed\n\n**kirannhegde** commented on 2023-09-21T11:40:14Z:\n\n> This is not working for me even with v2.8.4 version of Grafana Loki\n> \n> Loki version: v2.8.4\n> AKS version: 1.23.x\n> Storage backend: azure blob storage\n> \n> From reading the documenation for compactor, i get the impression that compactor is capable of deletion of old chunks from the blob storage.\n> However, i see that the old chunks of Loki are not getting deleted from the blob stroage even though the necessary configuration is in place.\n> Could somone be kind enough to tell me what could be wrong in my configuration?\n> When i inspect the logs for the compactor, i see that the marker file is being created. I also see a lot of API calls to: GET /loki/api/v1/delete.\n> However, i dont see any POST calls to /loki/api/v1/delete. So this gives me the impression that no deletion is happening now.\n> I confirmed the same that chunks from several months ago are still lying in my blob storage.\n> \n> ```\n> auth_enabled: false\n> \n> server:\n> http_listen_port: {{ .Values.loki.containerPorts.http }}\n> log_level: debug\n> common:\n> compactor_address: http://{{ include \"grafana-loki.compactor.fullname\" . }}:{{ .Values.compactor.service.ports.http }}\n> storage:\n> azure:\n> account_name: abc \n> account_key: abc\n> container_name: abc\n> use_managed_identity: false\n> request_timeout: 0 \n> \n> \n> distributor:\n> ring:\n> kvstore:\n> store: memberlist\n> \n> memberlist:\n> join_members:\n> - {{ include \"grafana-loki.gossip-ring.fullname\" . }}\n> \n> ingester:\n> lifecycler:\n> ring:\n> kvstore:\n> store: memberlist\n> replication_factor: 1\n> chunk_idle_period: 2h # Any chunk not receiving new logs in this time will be flushed\n> chunk_block_size: 262144\n> chunk_encoding: snappy\n> chunk_retain_period: 1m\n> max_chunk_age: 2h # All chunks will be flushed when they hit this age, default is 1h\n> max_transfer_retries: 0\n> autoforget_unhealthy: true\n> wal:\n> dir: {{ .Values.loki.dataDir }}/wal\n> \n> limits_config:\n> retention_period: 48h\n> enforce_metric_name: false\n> reject_old_samples: true\n> reject_old_samples_max_age: 168h\n> max_cache_freshness_per_query: 10m\n> split_queries_by_interval: 15m\n> per_stream_rate_limit: 10MB\n> per_stream_rate_limit_burst: 20MB\n> ingestion_rate_mb: 100\n> ingestion_burst_size_mb: 30\n> \n> schema_config:\n> configs:\n> - from: 2020-10-24\n> store: boltdb-shipper\n> object_store: azure\n> schema: v11\n> index:\n> prefix: index_\n> period: 24h\n> chunks:\n> period: 24h\n> \n> storage_config: \n> boltdb_shipper:\n> shared_store: azure\n> active_index_directory: {{ .Values.loki.dataDir }}/loki/index\n> cache_location: {{ .Values.loki.dataDir }}/loki/cache\n> cache_ttl: 168h\n> {{- if .Values.indexGateway.enabled }}\n> index_gateway_client:\n> server_address: {{ (printf \"dns:///%s:9095\" (include \"grafana-loki.index-gateway.fullname\" .)) }}\n> {{- end }}\n> filesystem:\n> directory: {{ .Values.loki.dataDir }}/chunks\n> index_queries_cache_config:\n> {{- if .Values.memcachedindexqueries.enabled }}\n> memcached:\n> batch_size: 100\n> parallelism: 100\n> memcached_client:\n> consistent_hash: true\n> addresses: dns+{{ include \"grafana-loki.memcached-index-queries.host\" . }}\n> service: http\n> {{- end }}\n> \n> chunk_store_config:\n> max_look_back_period: 2d\n> {{- if .Values.memcachedchunks.enabled }}\n> chunk_cache_config:\n> memcached:\n> batch_size: 100\n> parallelism: 100\n> memcached_client:\n> consistent_hash: true\n> addresses: dns+{{ include \"grafana-loki.memcached-chunks.host\" . }}\n> {{- end }}\n> {{- if .Values.memcachedindexwrites.enabled }}\n> write_dedupe_cache_config:\n> memcached:\n> batch_size: 100\n> parallelism: 100\n> memcached_client:\n> consistent_hash: true\n> addresses: dns+{{ include \"grafana-loki.memcached-index-writes.host\" . }}\n> {{- end }}\n> \n> table_manager:\n> retention_deletes_enabled: true\n> retention_period: 2d\n> \n> query_range:\n> align_queries_with_step: true\n> max_retries: 5\n> cache_results: true\n> results_cache:\n> cache:\n> {{- if .Values.memcachedfrontend.enabled }}\n> memcached_client:\n> consistent_hash: true\n> addresses: dns+{{ include \"grafana-loki.memcached-frontend.host\" . }}\n> max_idle_conns: 16\n> timeout: 500ms\n> update_interval: 1m\n> {{- else }}\n> enable_fifocache: true\n> fifocache:\n> max_size_items: 1024\n> validity: 24h\n> {{- end }}\n> {{- if not .Values.queryScheduler.enabled }}\n> frontend_worker:\n> frontend_address: {{ include \"grafana-loki.query-frontend.fullname\" . }}:{{ .Values.queryFrontend.service.ports.grpc }}\n> {{- end }}\n> \n> frontend:\n> log_queries_longer_than: 5s\n> compress_responses: true\n> tail_proxy_url: http://{{ include \"grafana-loki.querier.fullname\" . }}:{{ .Values.querier.service.ports.http }}\n> \n> compactor:\n> working_directory: {{ .Values.loki.dataDir }}/retention\n> shared_store: azure\n> compaction_interval: 10m\n> retention_enabled: true\n> retention_delete_delay: 2h\n> retention_delete_worker_count: 150\n> \n> ruler:\n> storage:\n> type: local\n> local:\n> directory: {{ .Values.loki.dataDir }}/conf/rules\n> ring:\n> kvstore:\n> store: memberlist\n> rule_path: /tmp/loki/scratch\n> alertmanager_url: http://abc.bdc.com/alertmanager\n> external_url: https://abc.bdc.com/alertmanager\n> ```\n\n**tomasz-kazmierczak** commented on 2023-09-30T19:39:45Z:\n\n> Hi,\n> I had a similar issue with recent 2.9.1 version of loki. It appears that recently there has been some work with the property `deletion_mode` (https://grafana.com/docs/loki/latest/operations/storage/logs-deletion/). This property is now a per tenant configurable within the runtime config, the default value is not documented but I had to enforce `filter-and-delete` mode and it started to delete chunks from my object storage.\n> \n> The helm values config for that setting is:\n> ```\n> loki:\n> runtimeConfig:\n> overrides:\n> fake:\n> deletion_mode: filter-and-delete\n> ```\n> I hope this solves your issues.\n\n- Referenced by issue #5: set a scheduled reminder to run every month\n\n**alexmeise** commented on 2024-01-03T09:39:14Z:\n\n> As far as I understood from https://github.com/grafana/loki/issues/7068#issuecomment-1347131945 chunks are never going to get deleted from s3 by the compactor. It is not possible to configure retention so that the compactor actually deletes the chunks from S3. This needs to be done via lifecyclePolicy or some other mechanism. What the compactor will do is manage the chunk/index relationship so that you dont receive issues for \"deleted chunks/indexes\" when running queries. I guess it will actually delete references to chunks in the indexes but not the chunk files themselves. Am I right?? It's a bit conter-intuitive and would help if this is clarified in the documentation. \n\n**szthanatos** commented on 2024-01-13T07:42:54Z:\n\n> > @Mastedont, I don't use S3, I store my chunks directly on disk. My current configuration:\n> > \n> > ```\n> > server:\n> > http_listen_port: 3100\n> > grpc_listen_port: 9096\n> > \n> > common:\n> > path_prefix: /var/lib/loki\n> > storage:\n> > filesystem:\n> > chunks_directory: /var/lib/loki/chunks\n> > rules_directory: /var/lib/loki/rules\n> > replication_factor: 1\n> > ring:\n> > instance_addr: 127.0.0.1\n> > kvstore:\n> > store: inmemory\n> > \n> > schema_config:\n> > configs:\n> > - from: 2020-10-24\n> > store: boltdb-shipper\n> > object_store: filesystem\n> > schema: v11\n> > index:\n> > prefix: index_\n> > period: 24h\n> > \n> > chunk_store_config:\n> > max_look_back_period: 168h\n> > \n> > compactor:\n> > working_directory: /var/lib/loki/retention\n> > shared_store: filesystem\n> > compaction_interval: 10m\n> > retention_enabled: true\n> > retention_delete_delay: 2h\n> > retention_delete_worker_count: 150\n> > \n> > limits_config:\n> > retention_period: 168h\n> > \n> > ruler:\n> > alertmanager_url: http://127.0.0.1:9093\n> > ```\n> \n> mark\n\n- Mastedont mentioned\n\n- Mastedont subscribed\n\n**otaku-bowei** commented on 2024-01-31T02:49:11Z:\n\n> > For everyone wondering what's going with the retention, I've tested the feature _a lot_ in the past days so here are what will work.\n> > \n> > ### Minimal Configuration Needed\n> > First of all, you absolutely need those config setup\n> > \n> > ```\n> > limits_config:\n> > retention_period: 10d # Keep 10 days\n> > compactor:\n> > delete_request_cancel_period: 10m # don't wait 24h before processing the delete_request\n> > retention_enabled: true # actually do the delete\n> > retention_delete_delay: 2h # wait 2 hours before actually deleting stuff\n> > ```\n> > \n> > You can tweak those config to delete faster or slower.\n> > \n> > ### Check If It's Working\n> > Once you got those config up and running, check that the logs are actually reporting that the retention is being applied : `msg=\"applying retention with compaction\"`. The \"caller\" for this log is `compactor.go`.\n> > \n> > Next, check that the retention manager is actually doing it's job in the logs: `msg=\"mark file created\"` and `msg=\"no marks file found\"` from the caller `marker.go`.\n> > \n> > The `mark file created` means that loki did found some chunks to be deleted and it has created a file to keep track of it. The `no marks file found` means that while performing the chunk delete routine, there was no file that matched it's filters, the filters mainly being the delay.\n> > \n> > Whenever you see the `mark file created` logs, you can go into the working directory of the compactor and check for the mark files. The path should be something like `/var/loki/compactor/retention/markers`. These files are kept there for 2 hours or whatever is set in `retention_delete_delay`. After `retention_delete_delay` is passed, loki _will_ delete the chunks.\n> > \n> > Not having any of the logs mentionned above means that the retention process is not started.\n> > \n> > ### Important Notes\n> > Loki will _only delete chunks that are indexed_. The indexes are actually being purged before deleting the chunks. This means that if you lose files from the compactor's working directory, whatever chunks that were marked there won't be deleted ever so it is still worth to have a lifecycle policy to cover for this OR have persistent storage for this particular folder.\n> \n> @nvanheuverzwijn learn a lot, how can i know which chunks are indexed?\n\n- nvanheuverzwijn mentioned\n\n- nvanheuverzwijn subscribed\n\n- Mastedont mentioned\n\n- Mastedont subscribed\n\n**myaswanth03** commented on 2024-05-30T07:15:04Z:\n\n> > I got the same issue. Logs older than 7 days are deleted and they're not visible in Grafana. Only the chunk files won't be deleted on the filesystem.\n> > \n> > ```\n> > loki, version 2.5.0 (branch: HEAD, revision: 2d9d0ee23)\n> > build user: root@4779f4b48f3a\n> > build date: 2022-04-07T21:50:00Z\n> > go version: go1.17.6\n> > platform: linux/amd64\n> > ```\n> > \n> > ```\n> > server:\n> > http_listen_port: 3100\n> > grpc_listen_port: 9096\n> > \n> > common:\n> > path_prefix: /var/lib/loki\n> > storage:\n> > filesystem:\n> > chunks_directory: /var/lib/loki/chunks\n> > rules_directory: /var/lib/loki/rules\n> > replication_factor: 1\n> > ring:\n> > instance_addr: 127.0.0.1\n> > kvstore:\n> > store: inmemory\n> > \n> > schema_config:\n> > configs:\n> > - from: 2020-10-24\n> > store: boltdb-shipper\n> > object_store: filesystem\n> > schema: v11\n> > index:\n> > prefix: index_\n> > period: 24h\n> > \n> > chunk_store_config:\n> > max_look_back_period: 168h\n> > \n> > table_manager:\n> > retention_deletes_enabled: true\n> > retention_period: 168h\n> > ```\n> \n> Hi, i used the same file, loki is up and running but retention is not working.\n> \n> permissions are also fine.\n> \n> \n> level=info ts=2024-05-30T07:12:05.009458014Z caller=metrics.go:159 component=frontend org_id=fake traceID=6cc7b11248ffd6f2 latency=fast query=\"sum by (level) (count_over_time({job=\\\"containerlogs\\\"} | drop __error__[1s]))\" query_hash=1978880347 query_type=metric range_type=range length=5m1s start_delta=5m2.009446978s end_delta=1.009448479s step=1s duration=930.872366ms status=200 limit=100 returned_lines=0 throughput=12kB total_bytes=11kB total_bytes_structured_metadata=0B lines_per_second=82 total_lines=77 post_filter_lines=77 total_entries=1 store_chunks_download_time=0s queue_time=100.027997ms splits=0 shards=16 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_chunk_download_time=0s cache_index_req=0 cache_index_hit=0 cache_index_download_time=0s cache_stats_results_req=0 cache_stats_results_hit=0 cache_stats_results_download_time=0s cache_result_req=0 cache_result_hit=0 cache_result_download_time=0s source=logvolhist\n> level=info ts=2024-05-30T07:12:05.180281106Z caller=roundtrip.go:241 org_id=fake traceID=7306103002fd206d msg=\"executing query\" type=range query=\"{job=\\\"containerlogs\\\"}\" length=5m0s step=5m0s query_hash=3605303691\n> level=info ts=2024-05-30T07:12:05.180937762Z caller=engine.go:234 component=querier org_id=fake traceID=7306103002fd206d msg=\"executing query\" type=range query=\"{job=\\\"containerlogs\\\"}\" length=5m0s step=5m0s query_hash=3605303691\n\n**krishgu** commented on 2024-08-08T20:12:56Z:\n\n> > > For everyone wondering what's going with the retention, I've tested the feature _a lot_ in the past days so here are what will work.\n> > > ### Minimal Configuration Needed\n> > > First of all, you absolutely need those config setup\n> > > ```\n> > > limits_config:\n> > > retention_period: 10d # Keep 10 days\n> > > compactor:\n> > > delete_request_cancel_period: 10m # don't wait 24h before processing the delete_request\n> > > retention_enabled: true # actually do the delete\n> > > retention_delete_delay: 2h # wait 2 hours before actually deleting stuff\n> > > ```\n> > > \n> > > \n> > > \n> > > \n> > > \n> > > \n> > > \n> > > \n> > > \n> > > \n> > > \n> > > You can tweak those config to delete faster or slower.\n> > > ### Check If It's Working\n> > > Once you got those config up and running, check that the logs are actually reporting that the retention is being applied : `msg=\"applying retention with compaction\"`. The \"caller\" for this log is `compactor.go`.\n> > > Next, check that the retention manager is actually doing it's job in the logs: `msg=\"mark file created\"` and `msg=\"no marks file found\"` from the caller `marker.go`.\n> > > The `mark file created` means that loki did found some chunks to be deleted and it has created a file to keep track of it. The `no marks file found` means that while performing the chunk delete routine, there was no file that matched it's filters, the filters mainly being the delay.\n> > > Whenever you see the `mark file created` logs, you can go into the working directory of the compactor and check for the mark files. The path should be something like `/var/loki/compactor/retention/markers`. These files are kept there for 2 hours or whatever is set in `retention_delete_delay`. After `retention_delete_delay` is passed, loki _will_ delete the chunks.\n> > > Not having any of the logs mentionned above means that the retention process is not started.\n> > > ### Important Notes\n> > > Loki will _only delete chunks that are indexed_. The indexes are actually being purged before deleting the chunks. This means that if you lose files from the compactor's working directory, whatever chunks that were marked there won't be deleted ever so it is still worth to have a lifecycle policy to cover for this OR have persistent storage for this particular folder.\n> > \n> > @nvanheuverzwijn learn a lot, how can i know which chunks are indexed?\n> \n> The above article by @nvanheuverzwijn is so on point. Thank you!\n> There are a lot of docs of setting Lifecycle Policies as the \"only\" way to purge, and those are outdated. This setup seems to be working fine. \n> \n> In the `compactor` or the `backend` pod, the `working_dir`/retention/`storage`/markers folder will have these marker files. \n> ```text\n> /var/loki/compactor/retention/aws/markers $ ls -la\n> total 312\n> drwxr-sr-x 2 loki loki 4096 Aug 8 19:50 .\n> drwxr-sr-x 3 loki loki 4096 Aug 7 19:59 ..\n> -rw-r--r-- 1 loki loki 32768 Aug 8 18:09 1723140578748104859\n> -rw-r--r-- 1 loki loki 32768 Aug 8 18:19 1723141178696907357\n> ...\n> -rw-r--r-- 1 loki loki 32768 Aug 8 19:49 1723146578638477113\n> -rw-r--r-- 1 loki loki 32768 Aug 8 19:59 1723147178354528360\n> ```\n> \n> If the compactor cycle is running every 10 minutes (default), you will see a new file created and the oldest file processed and deleted every 10 minutes. And in S3 or your storage providers these objects no longer exist!\n> \n> To find out what objects are about to be deleted, use the `strings` or `od -c` on the market file. So doing this on the oldest marker file will tell us about the next-in-line objects that are purged\n> \n> ```sh\n> /var/loki/compactor/retention/aws/markers $ strings 1723140578748104859\n> chunks\n> fake/65ff478cf0700326:190eb0ab615:190eb125142:ee83e028\n> fake/36bfb27dc259bf11:190eaa1db48:190eb0fef22:7101872a\n> fake/eecc8a1a1bba16ea:190eb0a8a31:190eb122448:156be59e\n> fake/dd094d6bff631bd5:190eaa581d0:190eb13688f:62c6862c\n> fake/f78ea3e78078c811:190eaa51fb0:190eb12fcfd:41bb1a13\n> fake/225757be61ef1d7a:190eaa50dad:190eb12f961:48b0c78c\n> fake/f4a7b9aef3c86224:190eb0c4cb2:190eb13a35e:a2919ec1\n> ```\n> \n> 1. To prove there is no old data, use the `aws s3api` command as below to list the oldest object.. You should not see any objects older than your retention_period if all went per plan\n> 2. you see that this is the oldest object is `fake/36bfb27dc259bf1...`, and this was in the output of the previous `strings 1723140578748104859` command (2nd of the fake/... objects). 10 minutes later you will see this deleted. This tallies correctly with the 14d retention period.\n> \n> ```sh\n> $ aws s3api list-objects-v2 --bucket dframe-loki- --prefix fake --query 'sort_by(Contents, &LastModified)[0]' --output json\n> {\n> \"Key\": \"fake/36bfb27dc259bf11:190eaa1db48:190eb0fef22:7101872a\",\n> \"LastModified\": \"2024-07-25T18:03:45.000Z\",\n> \"ETag\": \"\\\"606ec603a5cb748a96a3ea3e4fb11b87\\\"\",\n> \"Size\": 7000,\n> \"StorageClass\": \"STANDARD\"\n> \n> }\n> ```\n> \n> This ticket can be closed, not sure why it is still open...\n\n- nvanheuverzwijn mentioned\n\n- nvanheuverzwijn subscribed\n\n**kdvermagojoko** commented on 2024-08-14T07:20:00Z:\n\n> Hi, I have below conf in my helm override file for having retention period of 24hr, but still i see old index file in S3. I am using loki-stack-2.10.2 chart. Any idea what i am missing here?\n> \n> ```\n> loki:\n> serviceAccount:\n> name: loki-service-account\n> create: false\n> config:\n> schema_config:\n> configs:\n> - from: 2020-10-24\n> store: boltdb-shipper\n> object_store: s3\n> schema: v11\n> index:\n> prefix: loki_index_\n> period: 24h\n> storage_config:\n> aws:\n> s3: s3://aws_region/s3-bucket\n> s3forcepathstyle: true\n> bucketnames: s3-bucket\n> region: aws_region\n> insecure: false\n> sse_encryption: false\n> boltdb_shipper:\n> shared_store: s3\n> cache_ttl: 24h\n> limits_config:\n> enforce_metric_name: false\n> reject_old_samples: true\n> reject_old_samples_max_age: 24h\n> max_entries_limit_per_query: 5000\n> retention_period: 24h\n> max_query_lookback: 24h\n> compactor:\n> working_directory: /data/loki/boltdb-shipper-compactor\n> shared_store: filesystem\n> retention_enabled: true\n> delete_request_cancel_period: 24h\n> retention_delete_delay: 2h\n> retention_delete_worker_count: 150\n> compaction_interval: 24h\n> table_manager:\n> retention_deletes_enabled: true\n> retention_period: 24h\n> ```\n\n**younsl** commented on 2024-08-28T00:49:33Z:\n\n> Here's my `loki-distributed` chart configuration backed by S3 storage. Need to set `delete_request_store` in compactor config, too.\n> \n> > [!NOTE]\n> > `compactor.delete_request_store` should be set to configure the store for delete requests. This is required when retention is enabled. See loki's retention doc.\n> \n> ```yaml\n> # charts/loki-distributed/values.yaml\n> loki:\n> config: |\n> compactor:\n> shared_store: s3\n> working_directory: /var/loki/compactor\n> compaction_interval: 10m\n> retention_enabled: true\n> retention_delete_delay: 2h\n> retention_delete_worker_count: 150\n> delete_request_store: s3\n> \n> limits_config:\n> enforce_metric_name: false\n> reject_old_samples: true\n> reject_old_samples_max_age: 168h\n> max_cache_freshness_per_query: 10m\n> split_queries_by_interval: 15m\n> retention_period: 7d\n> ingestion_rate_mb: 20\n> ingestion_burst_size_mb: 30\n> ```\n> \n> ## Reference\n> \n> issue #9207\n\n- karakuraizer subscribed\n\n- karakuraizer unsubscribed\n\n- paulojmdias subscribed\n\n**adityamandke23** commented on 2024-12-30T07:12:33Z:\n\n> > @nvanheuverzwijn I do see mark file created but then a few mins later the logs show\n> > \n> > ```\n> > level=info ts=2023-09-11T05:23:41.019592474Z caller=compactor.go:364 msg=\"applying retention with compaction\"\n> > level=info ts=2023-09-11T05:23:41.020937376Z caller=expiration.go:60 msg=\"overall smallest retention period 1691817821.02, default smallest retention period 1691817821.02\"\n> > level=info ts=2023-09-11T05:23:41.065287615Z caller=marker.go:78 msg=\"mark file created\" file=/data/loki/boltdb-shipper-compactor/retention/markers/1694409821056046054\n> > ```\n> > \n> > Few secs later\n> > \n> > ```\n> > level=info ts=2023-09-11T05:24:41.019702032Z caller=marker.go:203 msg=\"no marks file found\"\n> > level=info ts=2023-09-11T05:25:41.020103387Z caller=marker.go:203 msg=\"no marks file found\"\n> > level=info ts=2023-09-11T05:26:41.020436386Z caller=marker.go:203 msg=\"no marks file found\"\n> > level=info ts=2023-09-11T05:27:41.020020721Z caller=marker.go:203 msg=\"no marks file found\"\n> > level=info ts=2023-09-11T05:28:41.01973026Z caller=marker.go:203 msg=\"no marks file found\"\n> > level=info ts=2023-09-11T05:30:41.020528317Z caller=marker.go:203 msg=\"no marks file found\"\n> > ```\n> > \n> > So looks like even after 5 mins (retention_delete_delay) it still cannot find the mark file. I verified that the mark file exists in that location Below is my loki config\n> > \n> > ```\n> > auth_enabled: false\n> > chunk_store_config:\n> > max_look_back_period: 0s\n> > compactor:\n> > delete_request_cancel_period: 10m\n> > retention_delete_delay: 5m\n> > retention_delete_worker_count: 150\n> > retention_enabled: true\n> > shared_store: filesystem\n> > working_directory: /data/loki/boltdb-shipper-compactor\n> > ingester:\n> > chunk_block_size: 262144\n> > chunk_idle_period: 3m\n> > chunk_retain_period: 1m\n> > lifecycler:\n> > ring:\n> > replication_factor: 1\n> > max_transfer_retries: 0\n> > wal:\n> > dir: /data/loki/wal\n> > limits_config:\n> > enforce_metric_name: false\n> > max_entries_limit_per_query: 5000\n> > max_query_lookback: 720h\n> > reject_old_samples: true\n> > reject_old_samples_max_age: 168h\n> > retention_period: 720h\n> > memberlist:\n> > join_members:\n> > - 'loki-memberlist'\n> > schema_config:\n> > configs:\n> > - from: \"2020-10-24\"\n> > index:\n> > period: 24h\n> > prefix: index_\n> > object_store: filesystem\n> > schema: v11\n> > store: boltdb-shipper\n> > server:\n> > grpc_listen_port: 9095\n> > http_listen_port: 3100\n> > http_server_read_timeout: 120s\n> > ```\n> \n> any fix for this? facing same issue.\n\n**rknightion** commented on 2025-01-26T19:17:33Z:\n\n> > Hi, I had a similar issue with recent 2.9.1 version of loki. It appears that recently there has been some work with the property `deletion_mode` (https://grafana.com/docs/loki/latest/operations/storage/logs-deletion/). This property is now a per tenant configurable within the runtime config, the default value is not documented but I had to enforce `filter-and-delete` mode and it started to delete chunks from my object storage.\n> > \n> > The helm values config for that setting is:\n> > \n> > ```\n> > loki:\n> > runtimeConfig:\n> > overrides:\n> > fake:\n> > deletion_mode: filter-and-delete\n> > ```\n> > \n> > I hope this solves your issues.\n> \n> THIS is what did it for me after following everything else in the thread. Despite a default limits_config being set to filter-and-delete I had to apply a per-tenant override for every tenant I have and now its actually deleting chunks!!\n\n- banschikovde subscribed\n\n**zzhura** commented on 2025-06-18T08:42:06Z:\n\n> sorry for offtopic but can comeone help with opposite problem - logs being deleted from my disk storage in ~11 days and no config changes helps :(\n> here's detailed description: https://community.grafana.com/t/logs-disappearing-after-11-days-in-storage/144618\n\n**Codecaver** commented on 2025-06-26T02:39:26Z:\n\n> > > Hi, I had a similar issue with recent 2.9.1 version of loki. It appears that recently there has been some work with the property `deletion_mode` (https://grafana.com/docs/loki/latest/operations/storage/logs-deletion/). This property is now a per tenant configurable within the runtime config, the default value is not documented but I had to enforce `filter-and-delete` mode and it started to delete chunks from my object storage.\n> > > The helm values config for that setting is:\n> > > ```\n> > > loki:\n> > > runtimeConfig:\n> > > overrides:\n> > > fake:\n> > > deletion_mode: filter-and-delete\n> > > ```\n> > > \n> > > \n> > > \n> > > \n> > > I hope this solves your issues.\n> > \n> > THIS is what did it for me after following everything else in the thread. Despite a default limits_config being set to filter-and-delete I had to apply a per-tenant override for every tenant I have and now its actually deleting chunks!!\n> \n> LGTM. and this is my config file:\n> Loki: 2.9.2\n> install method: Docker\n> ```\n> auth_enabled: false\n> \n> server:\n> http_listen_port: 3100\n> grpc_listen_port: 9096\n> \n> common:\n> instance_addr: 127.0.0.1\n> path_prefix: /tmp/loki\n> storage:\n> filesystem:\n> chunks_directory: /tmp/loki/chunks\n> rules_directory: /tmp/loki/rules\n> replication_factor: 1\n> ring:\n> kvstore:\n> store: inmemory\n> \n> compactor:\n> retention_enabled: true\n> retention_delete_delay: 0s\n> working_directory: /tmp/loki/compactor\n> #working_directory: /tmp/loki/chunks\n> shared_store: filesystem\n> \n> limits_config:\n> enforce_metric_name: false\n> reject_old_samples: true\n> reject_old_samples_max_age: 168h\n> ingestion_rate_mb: 50\n> ingestion_burst_size_mb: 100\n> per_stream_rate_limit: 256MB\n> per_stream_rate_limit_burst: 512MB\n> deletion_mode: filter-and-delete\n> retention_period: 168h\n> \n> table_manager:\n> retention_deletes_enabled: true\n> retention_period: 168h\n> \n> query_range:\n> results_cache:\n> cache:\n> embedded_cache:\n> enabled: true\n> max_size_mb: 1000\n> \n> schema_config:\n> configs:\n> - from: 2020-10-24\n> store: tsdb\n> object_store: filesystem\n> schema: v13\n> index:\n> prefix: index_\n> period: 24h\n> \n> ruler:\n> alertmanager_url: http://localhost:9093\n> ```\n\n**JStickler** commented on 2026-04-22T00:46:55Z:\n\n> Closing as stale. This issue was opened against a version of Loki that is no longer supported. If this issues still exists in more recent versions of Loki, please leave a comment and we can reopen the issue.\n\n- JStickler closed\n\n- ivankovnatsky unsubscribed", "content_chars": 56719, "published_date": null }, { "rank": 5, "url": "https://github.com/grafana/loki/blob/main/operator/internal/manifests/internal/config/loki-config.yaml", "title": "operator/internal/manifests/internal/config/loki-config.yaml at main \u00b7 grafana/loki", "content": "# operator/internal/manifests/internal/config/loki-config.yaml\n\n- Branch: main\n- Repository: grafana/loki\n\n---\n\n{{- /*gotype: github.com/grafana/loki/operator/internal/manifests/internal/config.Options*/ -}}\n---\nauth_enabled: true\nchunk_store_config:\n chunk_cache_config:\n embedded_cache:\n enabled: true\n max_size_mb: 500\ncommon:\n storage:\n object_store:\n {{- with .ObjectStorage.Azure }}\n azure:\n container_name: {{ .Container }}\n account_name: ${AZURE_STORAGE_ACCOUNT_NAME}\n endpoint_suffix: {{ .EndpointSuffix }}\n {{- if not .WorkloadIdentity }}\n account_key: ${AZURE_STORAGE_ACCOUNT_KEY}\n {{- end }}\n {{- end }}\n {{- with .ObjectStorage.GCS }}\n gcs:\n bucket_name: {{ .Bucket }}\n {{- end }}\n {{- with .ObjectStorage.S3 }}\n s3:\n {{- if .STS }}\n bucket_name: {{ .Buckets }}\n region: {{ .Region }}\n endpoint: s3.{{.Region}}.amazonaws.com\n native_aws_auth_enabled: true\n {{- else }}\n endpoint: {{ .Endpoint }}\n bucket_name: {{ .Buckets }}\n region: {{ .Region }}\n access_key_id: ${AWS_ACCESS_KEY_ID}\n secret_access_key: ${AWS_ACCESS_KEY_SECRET}\n {{- if .Insecure }}\n insecure: true\n {{- end}}\n {{- if .ForcePathStyle }}\n bucket_lookup_type: path\n {{- end}}\n {{- end }}\n {{- with .SSE }}\n {{- if .Type }}\n sse:\n type: {{ .Type }}\n {{- if eq .Type \"SSE-KMS\" }}\n kms_key_id: {{ .KMSKeyID }}\n {{- with .KMSEncryptionContext }}\n kms_encryption_context: |\n ${AWS_SSE_KMS_ENCRYPTION_CONTEXT}\n {{- end }}\n {{- end}}\n {{- end }}\n {{- end }}\n {{- end }}\n {{- with .ObjectStorage.Swift }}\n swift:\n auth_url: {{ .AuthURL }}\n username: ${SWIFT_USERNAME}\n user_domain_name: {{ .UserDomainName }}\n user_domain_id: {{ .UserDomainID }}\n user_id: {{ .UserID }}\n password: ${SWIFT_PASSWORD}\n domain_id: {{ .DomainID }}\n domain_name: {{ .DomainName }}\n project_id: {{ .ProjectID }}\n project_name: {{ .ProjectName }}\n project_domain_id: {{ .ProjectDomainID }}\n project_domain_name: {{ .ProjectDomainName }}\n region_name: {{ .Region }}\n container_name: {{ .Container }}\n {{- end }}\n {{- with .ObjectStorage.AlibabaCloud}}\n alibaba:\n bucket: {{ .Bucket }}\n endpoint: {{ .Endpoint }}\n access_key_id: ${ALIBABA_CLOUD_ACCESS_KEY_ID}\n access_key_secret: ${ALIBABA_CLOUD_ACCESS_KEY_SECRET}\n {{- end }}\n compactor_grpc_address: {{ .Compactor.FQDN }}:{{ .Compactor.Port }}\n {{- with .GossipRing }}\n ring:\n kvstore:\n store: memberlist\n heartbeat_period: 5s\n heartbeat_timeout: 1m\n {{- with .InstanceAddr }}\n instance_addr: {{ . }}\n {{- end }}\n instance_port: {{ .InstancePort }}\n {{- if .EnableInstanceAvailabilityZone }}\n zone_awareness_enabled: true\n instance_availability_zone: ${INSTANCE_AVAILABILITY_ZONE}\n {{- end }}\n {{- end }}\ncompactor:\n compaction_interval: 2h\n working_directory: {{ .StorageDirectory }}/compactor\n{{- if .Retention.Enabled }}\n{{- with .Retention }}\n retention_enabled: true\n retention_delete_delay: 4h\n retention_delete_worker_count: {{.DeleteWorkerCount}}\n{{- end }}\n delete_request_store: {{.ObjectStorage.SharedStore}}\n{{- end }}\n{{- if .OTLPAttributes.RemoveDefaultLabels }}\ndistributor:\n otlp_config:\n default_resource_attributes_as_index_labels: []\n{{- end }}\nfrontend:\n tail_proxy_url: {{ .Querier.Protocol }}://{{ .Querier.FQDN }}:{{ .Querier.Port }}\n{{- if .Gates.HTTPEncryption }}\n tail_tls_config:\n tls_cert_path: {{ .TLS.Paths.HTTP.Certificate }}\n tls_key_path: {{ .TLS.Paths.HTTP.Key }}\n tls_ca_path: {{ .TLS.Paths.CA }}\n tls_server_name: {{ .TLS.ServerNames.HTTP.Querier }}\n tls_cipher_suites: {{ .TLS.CipherSuitesString }}\n tls_min_version: {{ .TLS.MinTLSVersion }}\n{{- end }}\n compress_responses: true\n max_outstanding_per_tenant: 4096\n log_queries_longer_than: 5s\nfrontend_worker:\n frontend_address: {{ .FrontendWorker.FQDN }}:{{ .FrontendWorker.Port }}\n grpc_client_config:\n max_send_msg_size: 104857600\n{{- if .Gates.GRPCEncryption }}\n tls_enabled: true\n tls_cert_path: {{ .TLS.Paths.GRPC.Certificate }}\n tls_key_path: {{ .TLS.Paths.GRPC.Key }}\n tls_ca_path: {{ .TLS.Paths.CA }}\n tls_server_name: {{ .TLS.ServerNames.GRPC.QueryFrontend }}\n tls_cipher_suites: {{ .TLS.CipherSuitesString }}\n tls_min_version: {{ .TLS.MinTLSVersion }}\n{{- end }}\ningester:\n chunk_block_size: 262144\n chunk_encoding: snappy\n chunk_idle_period: 1h\n chunk_retain_period: 5m\n chunk_target_size: 2097152\n flush_op_timeout: 10m\n max_chunk_age: 2h\n autoforget_unhealthy: true\n lifecycler:\n final_sleep: 0s\n join_after: 30s\n num_tokens: 512\n tokens_file_path: {{ .StorageDirectory }}/ring-tokens\n {{- if .GossipRing.EnableIPv6 }}\n enable_inet6: true\n {{- end}}\n ring:\n replication_factor: {{ .Stack.Replication.Factor }}\n wal:\n enabled: true\n dir: {{ .WriteAheadLog.Directory }}\n replay_memory_ceiling: {{ .WriteAheadLog.ReplayMemoryCeiling }}\ningester_client:\n grpc_client_config:\n max_recv_msg_size: 67108864\n{{- if .Gates.GRPCEncryption }}\n tls_enabled: true\n tls_cert_path: {{ .TLS.Paths.GRPC.Certificate }}\n tls_key_path: {{ .TLS.Paths.GRPC.Key }}\n tls_ca_path: {{ .TLS.Paths.CA }}\n tls_server_name: {{ .TLS.ServerNames.GRPC.Ingester }}\n tls_cipher_suites: {{ .TLS.CipherSuitesString }}\n tls_min_version: {{ .TLS.MinTLSVersion }}\n{{- end }}\n remote_timeout: 5s\n# NOTE: Keep the order of keys as in Loki docs\n# to enable easy diffs when vendoring newer\n# Loki releases.\n# (See https://grafana.com/docs/loki/latest/configuration/#limits_config)\n#\n# Values for not exposed fields are taken from the grafana/loki production\n# configuration manifests.\n# (See https://github.com/grafana/loki/blob/main/production/ksonnet/loki/config.libsonnet)\nlimits_config:\n ingestion_rate_strategy: global\n ingestion_rate_mb: {{ .Stack.Limits.Global.IngestionLimits.IngestionRate }}\n ingestion_burst_size_mb: {{ .Stack.Limits.Global.IngestionLimits.IngestionBurstSize }}\n max_label_name_length: {{ .Stack.Limits.Global.IngestionLimits.MaxLabelNameLength }}\n max_label_value_length: {{ .Stack.Limits.Global.IngestionLimits.MaxLabelValueLength }}\n max_label_names_per_series: {{ .Stack.Limits.Global.IngestionLimits.MaxLabelNamesPerSeries }}\n reject_old_samples: true\n reject_old_samples_max_age: 168h\n creation_grace_period: 10m\n # Keep max_streams_per_user always to 0 to default\n # using max_global_streams_per_user always.\n # (See https://github.com/grafana/loki/blob/main/pkg/ingester/limiter.go#L73)\n max_streams_per_user: 0\n max_line_size: {{ .Stack.Limits.Global.IngestionLimits.MaxLineSize }}\n max_entries_limit_per_query: {{ .Stack.Limits.Global.QueryLimits.MaxEntriesLimitPerQuery }}\n discover_service_name: []\n discover_log_levels: {{ .DiscoverLogLevels }}\n max_global_streams_per_user: {{ .Stack.Limits.Global.IngestionLimits.MaxGlobalStreamsPerTenant }}\n max_chunks_per_query: {{ .Stack.Limits.Global.QueryLimits.MaxChunksPerQuery }}\n max_query_length: 721h\n max_query_parallelism: 32\n tsdb_max_query_parallelism: 512\n max_query_series: {{ .Stack.Limits.Global.QueryLimits.MaxQuerySeries }}\n cardinality_limit: {{ .Stack.Limits.Global.QueryLimits.CardinalityLimit }}\n max_streams_matchers_per_query: 1000\n query_timeout: {{ .Stack.Limits.Global.QueryLimits.QueryTimeout }}\n volume_enabled: true\n volume_max_series: {{ .Stack.Limits.Global.QueryLimits.MaxVolumeSeries }}\n{{- if .Retention.Enabled }}{{- with .Stack.Limits.Global.Retention }}\n retention_period: {{.Days}}d\n{{- with .Streams }}\n retention_stream:\n{{- range . }}\n - selector: '{{ .Selector }}'\n priority: {{ .Priority }}\n period: {{ .Days }}d\n{{- end }}\n{{- end }}\n{{- end }}{{- end }}\n max_cache_freshness_per_query: 10m\n per_stream_rate_limit: {{ .Stack.Limits.Global.IngestionLimits.PerStreamRateLimit }}MB\n per_stream_rate_limit_burst: {{ .Stack.Limits.Global.IngestionLimits.PerStreamRateLimitBurst }}MB\n split_queries_by_interval: 30m\n{{- with .Stack.Limits.Global.IngestionLimits.PerStreamDesiredRate }}\n shard_streams:\n enabled: true\n desired_rate: {{ . }}MB\n time_sharding_enabled: true\n{{- end }}\n{{- with .OTLPAttributes.Global }}\n otlp_config:\n {{- if .ResourceAttributes }}\n resource_attributes:\n {{- with .ResourceAttributes}}\n attributes_config:\n {{- range . }}\n - action: {{ .Action }}\n {{- with .Regex }}\n regex: {{ . }}\n {{- end }}\n {{- with .Names }}\n attributes:\n {{- range . }}\n - {{ . }}\n {{- end }}\n {{- end }}\n {{- end }}\n {{- end }}\n {{- end }}\n {{- with .ScopeAttributes }}\n scope_attributes:\n {{- range . }}\n - action: {{ .Action }}\n {{- with .Regex }}\n regex: {{ . }}\n {{- end }}\n {{- with .Names }}\n attributes:\n {{- range . }}\n - {{ . }}\n {{- end }}\n {{- end }}\n {{- end }}\n {{- end }}\n {{- with .LogAttributes }}\n log_attributes:\n {{- range . }}\n - action: {{ .Action }}\n {{- with .Regex }}\n regex: {{ . }}\n {{- end }}\n {{- with .Names }}\n attributes:\n {{- range . }}\n - {{ . }}\n {{- end }}\n {{- end }}\n {{- end }}\n {{- end }}\n{{- end }}\n allow_structured_metadata: {{ .ObjectStorage.AllowStructuredMetadata }}\n{{- with .GossipRing }}\nmemberlist:\n abort_if_cluster_join_fails: true\n rejoin_interval: 90s\n {{- with .InstanceAddr }}\n advertise_addr: {{ . }}\n {{- end }}\n advertise_port: {{ .BindPort }}\n bind_port: {{ .BindPort }}\n join_members:\n - {{ .MembersDiscoveryAddr }}:{{ .BindPort }}\n max_join_backoff: 1m\n max_join_retries: 10\n min_join_backoff: 1s\n{{- end }}\nquerier:\n engine:\n max_look_back_period: 30s\n extra_query_delay: 0s\n query_ingesters_within: 3h\n tail_max_duration: 1h\n max_concurrent: {{ .MaxConcurrent.AvailableQuerierCPUCores }}\n{{- if .Gates.GRPCEncryption }}\ncompactor_grpc_client:\n tls_enabled: true\n tls_cert_path: {{ .TLS.Paths.GRPC.Certificate }}\n tls_key_path: {{ .TLS.Paths.GRPC.Key }}\n tls_ca_path: {{ .TLS.Paths.CA }}\n tls_server_name: {{ .TLS.ServerNames.GRPC.Compactor }}\n tls_cipher_suites: {{ .TLS.CipherSuitesString }}\n tls_min_version: {{ .TLS.MinTLSVersion }}\n{{- end }}\nquery_range:\n align_queries_with_step: true\n cache_results: true\n max_retries: 5\n results_cache:\n cache:\n embedded_cache:\n enabled: true\n max_size_mb: 500\n parallelise_shardable_queries: true\nschema_config:\n configs:\n {{- $store := .ObjectStorage.SharedStore }}\n {{- range .ObjectStorage.Schemas }}\n - from: \"{{ .EffectiveDate }}\"\n index:\n period: 24h\n prefix: index_\n object_store: {{ $store }}\n schema: {{ .Version }}\n {{- if or (eq .Version \"v11\") (eq .Version \"v12\")}}\n store: boltdb-shipper\n {{- else }}\n store: tsdb\n {{- end}}\n {{- end }}\n{{ if .Ruler.Enabled }}\nruler_storage:\n backend: local\n local:\n directory: {{ .Ruler.RulesStorageDirectory }}\nruler:\n enable_api: true\n enable_sharding: true\n {{- with .Ruler.EvaluationInterval }}\n evaluation_interval: {{ . }}\n {{- end }}\n {{- with .Ruler.PollInterval }}\n poll_interval: {{ . }}\n {{- end }}\n {{- with .Ruler.AlertManager }}\n {{- with .ExternalURL }}\n external_url: {{ . }}\n {{- end}}\n {{- with .ExternalLabels }}\n external_labels:\n {{- range $name, $value := . }}\n {{ $name }}: {{ $value }}\n {{- end }}\n {{- end}}\n alertmanager_url: {{ .Hosts }}\n {{- if .EnableV2 }}\n enable_alertmanager_v2: {{ .EnableV2 }}\n {{- end }}\n {{- if .EnableDiscovery }}\n enable_alertmanager_discovery: {{ .EnableDiscovery }}\n alertmanager_refresh_interval: {{ .RefreshInterval }}\n {{- end }}\n {{- if .QueueCapacity }}\n notification_queue_capacity: {{ .QueueCapacity }}\n {{- end }}\n {{- if .Timeout }}\n notification_timeout: {{ .Timeout }}\n {{- end }}\n {{- if .Notifier }}\n {{- with .Notifier }}\n alertmanager_client:\n {{- if .TLS.CertPath }}\n tls_cert_path: {{ .TLS.CertPath }}\n {{- end }}\n {{- if .TLS.KeyPath }}\n tls_key_path: {{ .TLS.KeyPath }}\n {{- end }}\n {{- if .TLS.CAPath }}\n tls_ca_path: {{ .TLS.CAPath }}\n {{- end }}\n {{- if .TLS.ServerName }}\n tls_server_name: {{ .TLS.ServerName }}\n {{- end }}\n {{- if .TLS.InsecureSkipVerify }}\n tls_insecure_skip_verify: {{ .TLS.InsecureSkipVerify }}\n {{- end }}\n {{- if .TLS.CipherSuites }}\n tls_cipher_suites: {{ .TLS.CipherSuites }}\n {{- end }}\n {{- if .TLS.MinVersion }}\n tls_min_version: {{ .TLS.MinVersion }}\n {{- end }}\n {{- if .BasicAuth.Username }}\n basic_auth_username: {{ .BasicAuth.Username }}\n {{- end }}\n {{- if .BasicAuth.Password }}\n basic_auth_password: {{ .BasicAuth.Password }}\n {{- end }}\n {{- if .HeaderAuth.Type }}\n type: {{ .HeaderAuth.Type }}\n {{- end }}\n {{- if .HeaderAuth.Credentials }}\n credentials: {{ .HeaderAuth.Credentials }}\n {{- end }}\n {{- if .HeaderAuth.CredentialsFile }}\n credentials_file: {{ .HeaderAuth.CredentialsFile }}\n {{- end }}\n {{- end }}\n {{- end }}\n\n {{- with .RelabelConfigs }}\n alert_relabel_configs:\n {{- range $k, $cfg := . }}\n - {{ if $cfg.SourceLabels}}\n source_labels: {{ $cfg.SourceLabelsString }}\n {{- end }}\n {{- if $cfg.Regex }}\n regex: {{ $cfg.Regex }}\n {{- end }}\n {{- if $cfg.Action }}\n action: {{ $cfg.Action }}\n {{- end }}\n separator: {{ $cfg.SeparatorString }}\n {{- if $cfg.Replacement }}\n replacement: {{ $cfg.Replacement }}\n {{- end }}\n {{- if $cfg.TargetLabel }}\n target_label: {{ $cfg.TargetLabel }}\n {{- end }}\n {{- if $cfg.Modulus }}\n modulus: {{ $cfg.Modulus }}\n {{- end }}\n {{- end }}\n {{- end }}\n\n {{- if .ForOutageTolerance }}\n for_outage_tolerance: {{ .ForOutageTolerance }}\n {{- end }}\n {{- if .ForGracePeriod }}\n for_grace_period: {{ .ForGracePeriod }}\n {{- end }}\n {{- if .ResendDelay }}\n resend_delay: {{ .ResendDelay }}\n {{- end }}\n {{- end}}\n {{- with .Ruler.RemoteWrite }}\n {{- if .Enabled }}\n remote_write:\n enabled: {{ .Enabled }}\n config_refresh_period: {{ .RefreshPeriod }}\n clients:\n {{- with .Client }}\n default:\n name: {{ .Name }}\n url: {{ .URL }}\n remote_timeout: {{ .RemoteTimeout }}\n proxy_url: {{ .ProxyURL }}\n follow_redirects: {{ .FollowRedirects }}\n headers:\n {{- range $key, $val := .Headers }}\n \"{{ $key }}\": \"{{ $val }}\"\n {{- end }}\n {{- if and .BasicAuthUsername .BasicAuthPassword }}\n basic_auth:\n username: {{ .BasicAuthUsername }}\n password: {{ .BasicAuthPassword }}\n {{- end }}\n {{- if .BearerToken }}\n authorization:\n type: bearer\n credentials: {{ .BearerToken }}\n {{- end }}\n {{- end}}\n {{- with .RelabelConfigs }}\n write_relabel_configs:\n {{- range $k, $cfg := . }}\n - {{- if $cfg.SourceLabels}}\n source_labels: {{ $cfg.SourceLabelsString }}\n {{- end }}\n {{- if $cfg.Regex }}\n regex: {{ $cfg.Regex }}\n {{- end }}\n {{- if $cfg.Action }}\n action: {{ $cfg.Action }}\n {{- end }}\n separator: {{ $cfg.SeparatorString }}\n {{- if $cfg.Replacement }}\n replacement: {{ $cfg.Replacement }}\n {{- end }}\n {{- if $cfg.TargetLabel }}\n target_label: {{ $cfg.TargetLabel }}\n {{- end }}\n {{- if $cfg.Modulus }}\n modulus: {{ $cfg.Modulus }}\n {{- end }}\n {{- end}}\n {{- end }}\n {{- with .Queue }}\n queue_config:\n {{- if .Capacity }}\n capacity: {{ .Capacity }}\n {{- end }}\n {{- if .MaxShards }}\n max_shards: {{ .MaxShards }}\n {{- end }}\n {{- if .MinShards }}\n min_shards: {{ .MinShards }}\n {{- end }}\n {{- if .MaxSamplesPerSend }}\n max_samples_per_send: {{ .MaxSamplesPerSend }}\n {{- end }}\n {{- if .BatchSendDeadline }}\n batch_send_deadline: {{ .BatchSendDeadline }}\n {{- end }}\n {{- if .MinBackOffPeriod }}\n min_backoff: {{ .MinBackOffPeriod }}\n {{- end }}\n {{- if .MaxBackOffPeriod }}\n max_backoff: {{ .MaxBackOffPeriod }}\n {{- end }}\n {{- end }}\n {{- end }}\n {{- end }}\n wal:\n dir: {{ .WriteAheadLog.Directory }}\n truncate_frequency: 60m\n min_age: 5m\n max_age: 4h\n rule_path: {{ .StorageDirectory }}\n ring:\n kvstore:\n store: memberlist\n{{- if .Gates.GRPCEncryption }}\n ruler_client:\n tls_enabled: true\n tls_cert_path: {{ .TLS.Paths.GRPC.Certificate }}\n tls_key_path: {{ .TLS.Paths.GRPC.Key }}\n tls_ca_path: {{ .TLS.Paths.CA }}\n tls_server_name: {{ .TLS.ServerNames.GRPC.Ruler }}\n tls_cipher_suites: {{ .TLS.CipherSuitesString }}\n tls_min_version: {{ .TLS.MinTLSVersion }}\n{{- end }}\n{{ end }}\n{{- if .Gates.HTTPEncryption }}\ninternal_server:\n enable: true\n http_listen_address: \"\"\n tls_min_version: {{ .TLS.MinTLSVersion }}\n tls_cipher_suites: {{ .TLS.CipherSuitesString }}\n http_tls_config:\n cert_file: {{ .TLS.Paths.HTTP.Certificate }}\n key_file: {{ .TLS.Paths.HTTP.Key }}\n{{- end }}\nserver:\n graceful_shutdown_timeout: 5s\n grpc_server_min_time_between_pings: '10s'\n grpc_server_ping_without_stream_allowed: true\n grpc_server_max_concurrent_streams: 1000\n grpc_server_max_recv_msg_size: 104857600\n grpc_server_max_send_msg_size: 104857600\n http_listen_port: 3100\n http_server_idle_timeout: {{ .HTTPTimeouts.IdleTimeout }}\n http_server_read_timeout: {{ .HTTPTimeouts.ReadTimeout }}\n http_server_write_timeout: {{ .HTTPTimeouts.WriteTimeout }}\n{{- if or .Gates.HTTPEncryption .Gates.GRPCEncryption }}\n tls_min_version: {{ .TLS.MinTLSVersion }}\n tls_cipher_suites: {{ .TLS.CipherSuitesString }}\n{{- if .Gates.HTTPEncryption }}\n http_tls_config:\n cert_file: {{ .TLS.Paths.HTTP.Certificate }}\n key_file: {{ .TLS.Paths.HTTP.Key }}\n client_auth_type: RequireAndVerifyClientCert\n client_ca_file: {{ .TLS.Paths.CA }}\n{{- end }}\n{{- if .Gates.GRPCEncryption }}\n grpc_tls_config:\n cert_file: {{ .TLS.Paths.GRPC.Certificate }}\n key_file: {{ .TLS.Paths.GRPC.Key }}\n client_auth_type: RequireAndVerifyClientCert\n client_ca_file: {{ .TLS.Paths.CA }}\n{{- end }}\n{{- end }}\n log_level: info\nstorage_config:\n use_thanos_objstore: true\n{{- range $_, $ship := .Shippers }}\n{{- if eq $ship \"boltdb\" }}\n boltdb_shipper:\n active_index_directory: {{ $.StorageDirectory }}/index\n cache_location: {{ $.StorageDirectory }}/index_cache\n{{- end }}\n{{- if eq $ship \"tsdb\" }}\n tsdb_shipper:\n active_index_directory: {{ $.StorageDirectory }}/tsdb-index\n cache_location: {{ $.StorageDirectory }}/tsdb-cache\n{{- end }}\n cache_ttl: 24h\n resync_interval: 5m\n index_gateway_client:\n server_address: dns:///{{ $.IndexGateway.FQDN }}:{{ $.IndexGateway.Port }}\n{{- if $.Gates.GRPCEncryption }}\n grpc_client_config:\n tls_enabled: true\n tls_cert_path: {{ $.TLS.Paths.GRPC.Certificate }}\n tls_key_path: {{ $.TLS.Paths.GRPC.Key }}\n tls_ca_path: {{ $.TLS.Paths.CA }}\n tls_server_name: {{ $.TLS.ServerNames.GRPC.IndexGateway }}\n tls_cipher_suites: {{ $.TLS.CipherSuitesString }}\n tls_min_version: {{ $.TLS.MinTLSVersion }}\n{{- end }}\n{{- end }}\ntracing:\n enabled: false\nanalytics:\n reporting_enabled: {{ .EnableRemoteReporting }}\n", "content_chars": 17751, "published_date": null }, { "rank": 6, "url": "https://grafana.com/docs/loki/v2.6.x/operations/storage/retention", "title": "Grafana Loki Storage Retention", "content": "Retention | Grafana Loki documentation\n\nTry using Grot AI for this query ->\n\nMenu\n\nImportant: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.\n\nDocumentation Grafana Loki Operations Storage Retention\n\nOpen source\n\n# Grafana Loki Storage Retention\n\nRetention in Grafana Loki is achieved either through the Table Manager or the Compactor.\n\nRetention through the Table Manager is achieved by relying on the object store TTL feature, and will work for both boltdb-shipper store and chunk/index store. However retention through the Compactor is supported only with the boltdb-shipper store.\n\nThe Compactor retention will become the default and have long term support. It supports more granular retention policies on per tenant and per stream use cases.\n\n## Compactor\n\nThe Compactor can deduplicate index entries. It can also apply granular retention. When applying retention with the Compactor, the Table Manager is unnecessary.\n\nRun the compactor as a singleton (a single instance).\n\nCompaction and retention are idempotent. If the compactor restarts, it will continue from where it left off.\n\nThe Compactor loops to apply compaction and retention at every`compaction_interval`, or as soon as possible if running behind.\n\nThe compactor\u2019s algorithm to update the index:\n\nFor each table within each day:\n\n- Compact the table into a single index file.\n- Traverse the entire index. Use the tenant configuration to identify and mark chunks that need to be removed.\n- Remove marked chunks from the index and save their reference in a file on disk.\n- Upload the new modified index files.\n\nThe retention algorithm is applied to the index. Chunks are not deleted while applying the retention algorithm. The chunks will be deleted by the compactor asynchronously when swept.\n\nMarked chunks will only be deleted after`retention_delete_delay` configured is expired because:\n\nboltdb-shipper indexes are refreshed from the shared store on components using it (querier and ruler) at a specific interval. This means deleting chunks instantly could lead to components still having reference to old chunks and so they could fails to execute queries. Having a delay allows for components to refresh their store and so remove gracefully their reference of those chunks.\n\nIt provides a short window of time in which to cancel chunk deletion in the case of a configuration mistake.\n\nMarker files (containing chunks to delete) should be stored on a persistent disk, since the disk will be the sole reference to them.\n\n### Retention Configuration\n\nThis compactor configuration example activates retention.\n\nYAML Copy\n\n```\ncompactor:\n working_directory: /data/retention\n shared_store: gcs\n compaction_interval: 10m\n retention_enabled: true\n retention_delete_delay: 2h\n retention_delete_worker_count: 150\nschema_config:\n configs:\n - from: \"2020-07-31\"\n index:\n period: 24h\n prefix: loki_index_\n object_store: gcs\n schema: v11\n store: boltdb-shipper\nstorage_config:\n boltdb_shipper:\n active_index_directory: /data/index\n cache_location: /data/boltdb-cache\n shared_store: gcs\n gcs:\n bucket_name: loki\n```\n\nNote that retention is only available if the index period is 24h.\n\nSet`retention_enabled` to true. Without this, the Compactor will only compact tables.\n\nDefine`schema_config` and`storage_config` to access the storage.\n\nThe index period must be 24h.\n\n`working_directory` is the directory where marked chunks and temporary tables will be saved.\n\n`compaction_interval` dictates how often compaction and/or retention is applied. If the Compactor falls behind, compaction and/or retention occur as soon as possible.\n\n`retention_delete_delay` is the delay after which the compactor will delete marked chunks.\n\n`retention_delete_worker_count` specifies the maximum quantity of goroutine workers instantiated to delete chunks.\n\n#### Configuring the retention period\n\nRetention period is configured within the limits_config configuration section.\n\nThere are two ways of setting retention policies:\n\n- `retention_period` which is applied globally.\n- `retention_stream` which is only applied to chunks matching the selector\n\nThe minimum retention period is 24h.\n\nThis example configures global retention:\n\nYAML Copy\n\n```\n...\nlimits_config:\n retention_period: 744h\n retention_stream:\n - selector: '{namespace=\"dev\"}'\n priority: 1\n period: 24h\n per_tenant_override_config: /etc/overrides.yaml\n...\n```\n\nNOTE: You can only use label matchers in the`selector` field of a`retention_stream` definition. Arbitrary LogQL expressions are not supported.\n\nPer tenant retention can be defined using the`/etc/overrides.yaml` files. For example:\n\nYAML Copy\n\n```\noverrides:\n \"29\":\n retention_period: 168h\n retention_stream:\n - selector: '{namespace=\"prod\", container=~\"(nginx|loki)\"}'\n priority: 2\n period: 336h\n - selector: '{container=\"loki\"}'\n priority: 1\n period: 72h\n \"30\":\n retention_stream:\n - selector: '{container=\"nginx\", level=\"debug\"}'\n priority: 1\n period: 24h\n```\n\nA rule to apply is selected by choosing the first in this list that matches:\n\n1. If a per-tenant`retention_stream` matches the current stream, the highest priority is picked.\n2. If a global`retention_stream` matches the current stream, the highest priority is picked.\n3. If a per-tenant`retention_period` is specified, it will be applied.\n4. The global`retention_period` will be selected if nothing else matched.\n5. If no global`retention_period` is specified, the default value of`744h`(30days) retention is used.\n\nStream matching uses the same syntax as Prometheus label matching:\n\n- `=`: Select labels that are exactly equal to the provided string.\n- `!=`: Select labels that are not equal to the provided string.\n- `=~`: Select labels that regex-match the provided string.\n- `!~`: Select labels that do not regex-match the provided string.\n\nThe example configurations will set these rules:\n\nFor tenant`29`:\n\n- All streams except those in the container`loki` or in the namespace`prod` will have retention period of`168h`(1 week).\n- All streams in the`prod` namespace will have a retention period of`336h`(2 weeks), even if the container label is`loki`, since the priority of the`prod` rule is higher.\n- Streams that have the container label`loki` but are not in the namespace`prod` will have a`72h` retention period.\n\nFor tenant`30`:\n\n- All streams except those having the container label`nginx` will have the global retention period of`744h`, since there is no override specified.\n- Streams that have the label`nginx` will have a retention period of`24h`.\n\n## Table Manager\n\nIn order to enable the retention support, the Table Manager needs to be configured to enable deletions and a retention period. Please refer to the table_manager section of the Loki configuration reference for all available options. Alternatively, the`table-manager.retention-period` and`table-manager.retention-deletes-enabled` command line flags can be used. The provided retention period needs to be a duration represented as a string that can be parsed using the Prometheus common model ParseDuration. Examples:`7d`,`1w`,`168h`.\n\nWARNING: The retention period must be a multiple of the index and chunks table`period`, configured in the period_config block. See the Table Manager documentation for more information.\n\nNOTE: To avoid querying of data beyond the retention period,`max_look_back_period` config in chunk_store_config must be set to a value less than or equal to what is set in`table_manager.retention_period`.\n\nWhen using S3 or GCS, the bucket storing the chunks needs to have the expiry policy set correctly. For more details check S3\u2019s documentation or GCS\u2019s documentation.\n\nCurrently, the retention policy can only be set globally. A per-tenant retention policy with an API to delete ingested logs is still under development.\n\nSince a design goal of Loki is to make storing logs cheap, a volume-based deletion API is deprioritized. Until this feature is released, if you suddenly must delete ingested logs, you can delete old chunks in your object store. Note, however, that this only deletes the log content and keeps the label index intact; you will still be able to see related labels but will be unable to retrieve the deleted log content.\n\nFor further details on the Table Manager internals, refer to the Table Manager documentation.\n\n## Example Configuration\n\nExample configuration with GCS with a 28 day retention:\n\nYAML Copy\n\n```\nschema_config:\n configs:\n - from: 2018-04-15\n store: bigtable\n object_store: gcs\n schema: v11\n index:\n prefix: loki_index_\n period: 168h\n\nstorage_config:\n bigtable:\n instance: BIGTABLE_INSTANCE\n project: BIGTABLE_PROJECT\n gcs:\n bucket_name: GCS_BUCKET_NAME\n\nchunk_store_config:\n max_look_back_period: 672h\n\ntable_manager:\n retention_deletes_enabled: true\n retention_period: 672h\n```\n\n## Was this page helpful?\n\nYes No\n\nSuggest an edit in GitHub\n\nCreate a GitHub issue\n\nEmail docs@grafana.com\n\nHelp and support\n\nCommunity\n\n## Related resources from Grafana Labs\n\nAdditional helpful documentation, links, and articles:", "content_chars": 9362, "published_date": null }, { "rank": 7, "url": "https://github.com/grafana/loki/blob/v1.3.0/docs/configuration/README.md", "title": "docs/configuration/README.md", "content": "# docs/configuration/README.md\n\n- Branch: v1.3.0\n- Repository: grafana/loki\n\n---\n\n# Configuring Loki\n\nLoki is configured in a YAML file (usually referred to as `loki.yaml`)\nwhich contains information on the Loki server and its individual components,\ndepending on which mode Loki is launched in.\n\nConfiguration examples can be found in the Configuration Examples document.\n\n* Configuration File Reference\n* server_config\n* distributor_config\n* querier_config\n* ingester_client_config\n * grpc_client_config\n* ingester_config\n * lifecycler_config\n * ring_config\n* storage_config\n * cache_config\n* chunk_store_config\n* schema_config\n * period_config\n* limits_config\n* table_manager_config\n * provision_config\n * auto_scaling_config\n\n## Configuration File Reference\n\nTo specify which configuration file to load, pass the `-config.file` flag at the\ncommand line. The file is written in YAML format,\ndefined by the scheme below. Brackets indicate that a parameter is optional. For\nnon-list parameters the value is set to the specified default.\n\nGeneric placeholders are defined as follows:\n\n* ` `: a boolean that can take the values `true` or `false`\n* ` `: any integer matching the regular expression `[1-9]+[0-9]*`\n* ` `: a duration matching the regular expression `[0-9]+(ns|us|\u00b5s|ms|[smh])`\n* ` `: a string matching the regular expression `[a-zA-Z_][a-zA-Z0-9_]*`\n* ` `: a string of unicode characters\n* ` `: a valid path relative to current working directory or an\n absolute path.\n* ` `: a valid string consisting of a hostname or IP followed by an optional port number\n* ` `: a regular string\n* ` `: a regular string that is a secret, such as a password\n\nSupported contents and default values of `loki.yaml`:\n\n```yaml\n# The module to run Loki with. Supported values\n# all, querier, table-manager, ingester, distributor\n[target: | default = \"all\"]\n\n# Enables authentication through the X-Scope-OrgID header, which must be present\n# if true. If false, the OrgID will always be set to \"fake\".\n[auth_enabled: | default = true]\n\n# Configures the server of the launched module(s).\n[server: ]\n\n# Configures the distributor.\n[distributor: ]\n\n# Configures the querier. Only appropriate when running all modules or\n# just the querier.\n[querier: ]\n\n# Configures how the distributor will connect to ingesters. Only appropriate\n# when running all modules, the distributor, or the querier.\n[ingester_client: ]\n\n# Configures the ingester and how the ingester will register itself to a\n# key value store.\n[ingester: ]\n\n# Configures where Loki will store data.\n[storage_config: ]\n\n# Configures how Loki will store data in the specific store.\n[chunk_store_config: ]\n\n# Configures the chunk index schema and where it is stored.\n[schema_config: ]\n\n# Configures limits per-tenant or globally\n[limits_config: ]\n\n# Configures the table manager for retention\n[table_manager: ]\n```\n\n## server_config\n\nThe `server_config` block configures Promtail's behavior as an HTTP server:\n\n```yaml\n# HTTP server listen host\n[http_listen_address: ]\n\n# HTTP server listen port\n[http_listen_port: | default = 80]\n\n# gRPC server listen host\n[grpc_listen_address: ]\n\n# gRPC server listen port\n[grpc_listen_port: | default = 9095]\n\n# Register instrumentation handlers (/metrics, etc.)\n[register_instrumentation: | default = true]\n\n# Timeout for graceful shutdowns\n[graceful_shutdown_timeout: | default = 30s]\n\n# Read timeout for HTTP server\n[http_server_read_timeout: | default = 30s]\n\n# Write timeout for HTTP server\n[http_server_write_timeout: | default = 30s]\n\n# Idle timeout for HTTP server\n[http_server_idle_timeout: | default = 120s]\n\n# Max gRPC message size that can be received\n[grpc_server_max_recv_msg_size: | default = 4194304]\n\n# Max gRPC message size that can be sent\n[grpc_server_max_send_msg_size: | default = 4194304]\n\n# Limit on the number of concurrent streams for gRPC calls (0 = unlimited)\n[grpc_server_max_concurrent_streams: | default = 100]\n\n# Log only messages with the given severity or above. Supported values [debug,\n# info, warn, error]\n[log_level: | default = \"info\"]\n\n# Base path to server all API routes from (e.g., /v1/).\n[http_path_prefix: ]\n```\n\n## distributor_config\n\nThe `distributor_config` block configures the Loki Distributor.\n\n```yaml\n# Configures the distributors ring, used when the \"global\" ingestion rate\n# strategy is enabled.\n[ring: ]\n```\n\n## querier_config\n\nThe `querier_config` block configures the Loki Querier.\n\n```yaml\n# Timeout when querying ingesters or storage during the execution of a\n# query request.\n[query_timeout: | default = 1m]\n\n# Limit of the duration for which live tailing requests should be\n# served.\n[tail_max_duration: | default = 1h]\n\n# Time to wait before sending more than the minimum successful query\n# requests.\n[extra_query_delay: | default = 0s]\n\n# Configuration options for the LogQL engine.\nengine:\n # Timeout for query execution\n [timeout: | default = 3m]\n\n # The maximum amount of time to look back for log lines. Only\n # applicable for instant log queries.\n [max_look_back_period: | default = 30s]\n```\n\n## ingester_client_config\n\nThe `ingester_client_config` block configures how connections to ingesters\noperate.\n\n```yaml\n# Configures how connections are pooled\npool_config:\n # Whether or not to do health checks.\n [health_check_ingesters: | default = false]\n\n # How frequently to clean up clients for servers that have gone away after\n # a health check.\n [client_cleanup_period: | default = 15s]\n\n # How quickly a dead client will be removed after it has been detected\n # to disappear. Set this to a value to allow time for a secondary\n # health check to recover the missing client.\n [remotetimeout: ]\n\n# The remote request timeout on the client side.\n[remote_timeout: | default = 5s]\n\n# Configures how the gRPC connection to ingesters work as a\n# client.\n[grpc_client_config: ]\n```\n\n### grpc_client_config\n\nThe `grpc_client_config` block configures a client connection to a gRPC service.\n\n```yaml\n# The maximum size in bytes the client can receive\n[max_recv_msg_size: | default = 104857600]\n\n# The maximum size in bytes the client can send\n[max_send_msg_size: | default = 16777216]\n\n# Whether or not messages should be compressed\n[use_gzip_compression: | default = false]\n\n# Rate limit for gRPC client. 0 is disabled\n[rate_limit: | default = 0]\n\n# Rate limit burst for gRPC client.\n[rate_limit_burst: | default = 0]\n\n# Enable backoff and retry when a rate limit is hit.\n[backoff_on_ratelimits: | default = false]\n\n# Configures backoff when enbaled.\nbackoff_config:\n # Minimum delay when backing off.\n [minbackoff: | default = 100ms]\n\n # The maximum delay when backing off.\n [maxbackoff: | default = 10s]\n\n # Number of times to backoff and retry before failing.\n [maxretries: | default = 10]\n```\n\n## ingester_config\n\nThe `ingester_config` block configures Ingesters.\n\n```yaml\n# Configures how the lifecycle of the ingester will operate\n# and where it will register for discovery.\n[lifecycler: ]\n\n# Number of times to try and transfer chunks when leaving before\n# falling back to flushing to the store. Zero = no transfers are done.\n[max_transfer_retries: | default = 10]\n\n# How many flushes can happen concurrently from each stream.\n[concurrent_flushes: | default = 16]\n\n# How often should the ingester see if there are any blocks\n# to flush\n[flush_check_period: | default = 30s]\n\n# The timeout before a flush is cancelled\n[flush_op_timeout: | default = 10s]\n\n# How long chunks should be retained in-memory after they've\n# been flushed.\n[chunk_retain_period: | default = 15m]\n\n# How long chunks should sit in-memory with no updates before\n# being flushed if they don't hit the max block size. This means\n# that half-empty chunks will still be flushed after a certain\n# period as long as they receieve no further activity.\n[chunk_idle_period: | default = 30m]\n\n# The targeted _uncompressed_ size in bytes of a chunk block\n# When this threshold is exceeded the head block will be cut and compressed inside the chunk\n[chunk_block_size: | default = 262144]\n\n# A target _compressed_ size in bytes for chunks.\n# This is a desired size not an exact size, chunks may be slightly bigger\n# or significantly smaller if they get flushed for other reasons (e.g. chunk_idle_period)\n# The default value of 0 for this will create chunks with a fixed 10 blocks,\n# A non zero value will create chunks with a variable number of blocks to meet the target size.\n[chunk_target_size: | default = 0]\n\n# The compression algorithm to use for chunks. (supported: gzip, lz4, snappy)\n# You should choose your algorithm depending on your need:\n# - `gzip` highest compression ratio but also slowest decompression speed. (144 kB per chunk)\n# - `lz4` fastest compression speed (188 kB per chunk)\n# - `snappy` fast and popular compression algorithm (272 kB per chunk)\n[chunk_encoding: | default = gzip]\n\n# Parameters used to synchronize ingesters to cut chunks at the same moment.\n# Sync period is used to roll over incoming entry to a new chunk. If chunk's utilization\n# isn't high enough (eg. less than 50% when sync_min_utilization is set to 0.5), then\n# this chunk rollover doesn't happen.\n[sync_period: | default = 0]\n[sync_min_utilization: | Default = 0]\n\n# The maximum number of errors a stream will report to the user\n# when a push fails. 0 to make unlimited.\n[max_returned_stream_errors: | default = 10]\n```\n\n### lifecycler_config\n\nThe `lifecycler_config` is used by the Ingester to control how that ingester\nregisters itself into the ring and manages its lifecycle during its stay in the\nring.\n\n```yaml\n# Configures the ring the lifecycler connects to\n[ring: ]\n\n# The number of tokens the lifecycler will generate and put into the ring if\n# it joined without transferring tokens from another lifecycler.\n[num_tokens: | default = 128]\n\n# Period at which to heartbeat to the underlying ring.\n[heartbeat_period: | default = 5s]\n\n# How long to wait to claim tokens and chunks from another member when\n# that member is leaving. Will join automatically after the duration expires.\n[join_after: | default = 0s]\n\n# Minimum duration to wait before becoming ready. This is to work around race\n# conditions with ingesters exiting and updating the ring.\n[min_ready_duration: | default = 1m]\n\n# Store tokens in a normalised fashion to reduce the number of allocations.\n[normalise_tokens: | default = false]\n\n# Name of network interfaces to read addresses from.\ninterface_names:\n - [ ... | default = [\"eth0\", \"en0\"]]\n\n# Duration to sleep before exiting to ensure metrics are scraped.\n[final_sleep: | default = 30s]\n```\n\n### ring_config\n\nThe `ring_config` is used to discover and connect to Ingesters.\n\n```yaml\nkvstore:\n # The backend storage to use for the ring. Supported values are\n # consul, etcd, inmemory\n store: \n\n # The prefix for the keys in the store. Should end with a /.\n [prefix: | default = \"collectors/\"]\n\n # Configuration for a Consul client. Only applies if store\n # is \"consul\"\n consul:\n # The hostname and port of Consul.\n [host: | duration = \"localhost:8500\"]\n\n # The ACL Token used to interact with Consul.\n [acltoken: ]\n\n # The HTTP timeout when communicating with Consul\n [httpclienttimeout: | default = 20s]\n\n # Whether or not consistent reads to Consul are enabled.\n [consistentreads: | default = true]\n\n # Configuration for an ETCD v3 client. Only applies if\n # store is \"etcd\"\n etcd:\n # The ETCD endpoints to connect to.\n endpoints:\n - \n\n # The Dial timeout for the ETCD connection.\n [dial_tmeout: | default = 10s]\n\n # The maximum number of retries to do for failed ops to ETCD.\n [max_retries: | default = 10]\n\n# The heartbeat timeout after which ingesters are skipped for\n# reading and writing.\n[heartbeat_timeout: | default = 1m]\n\n# The number of ingesters to write to and read from. Must be at least\n# 1.\n[replication_factor: | default = 3]\n```\n\n## storage_config\n\nThe `storage_config` block configures one of many possible stores for both the\nindex and chunks. Which configuration is read from depends on the schema_config\nblock and what is set for the store value.\n\n```yaml\n# Configures storing chunks in AWS. Required options only required when aws is\n# present.\naws:\n # S3 or S3-compatible URL to connect to. If only region is specified as a\n # host, the proper endpoint will be deduced. Use inmemory:/// to\n # use a mock in-memory implementation.\n s3: \n\n # Set to true to force the request to use path-style addressing\n [s3forcepathstyle: | default = false]\n\n # Configure the DynamoDB connection\n dynamodbconfig:\n # URL for DynamoDB with escaped Key and Secret encoded. If only region is specified as a\n # host, the proper endpoint will be deduced. Use inmemory:/// to\n # use a mock in-memory implementation.\n dynamodb: \n\n # DynamoDB table management requests per-second limit.\n [apilimit: | default = 2.0]\n\n # DynamoDB rate cap to back off when throttled.\n [throttlelimit: | default = 10.0]\n\n # Application Autoscaling endpoint URL with escaped Key and Secret\n # encoded.\n [applicationautoscaling: ]\n\n # Metics-based autoscaling configuration.\n metrics:\n # Use metrics-based autoscaling via this Prometheus query URL.\n [url: ]\n\n # Queue length above which we will scale up capacity.\n [targetqueuelen: | default = 100000]\n\n # Scale up capacity by this multiple\n [scaleupfactor: | default = 1.3]\n\n # Ignore throttling below this level (rate per second)\n [minthrottling: | default = 1]\n\n # Query to fetch ingester queue length\n [queuelengthquery: | default = \"sum(avg_over_time(cortex_ingester_flush_queue_length{job=\"cortex/ingester\"}[2m]))\"]\n\n # Query to fetch throttle rates per table\n [throttlequery: | default = \"sum(rate(cortex_dynamo_throttled_total{operation=\"DynamoDB.BatchWriteItem\"}[1m])) by (table) > 0\"]\n\n # Quer to fetch write capacity usage per table\n [usagequery: | default = \"sum(rate(cortex_dynamo_consumed_capacity_total{operation=\"DynamoDB.BatchWriteItem\"}[15m])) by (table) > 0\"]\n\n # Query to fetch read capacity usage per table\n [readusagequery: | default = \"sum(rate(cortex_dynamo_consumed_capacity_total{operation=\"DynamoDB.QueryPages\"}[1h])) by (table) > 0\"]\n\n # Query to fetch read errors per table\n [readerrorquery: | default = \"sum(increase(cortex_dynamo_failures_total{operation=\"DynamoDB.QueryPages\",error=\"ProvisionedThroughputExceededException\"}[1m])) by (table) > 0\"]\n\n # Number of chunks to group together to parallelise fetches (0 to disable)\n [chunkgangsize: | default = 10]\n\n # Max number of chunk get operations to start in parallel.\n [chunkgetmaxparallelism: | default = 32]\n\n# Configures storing chunks in Bigtable. Required fields only required\n# when bigtable is defined in config.\nbigtable:\n # BigTable project ID\n project: \n\n # BigTable instance ID\n instance: \n\n # Configures the gRPC client used to connect to Bigtable.\n [grpc_client_config: ]\n\n# Configures storing index in GCS. Required fields only required\n# when gcs is defined in config.\ngcs:\n # Name of GCS bucket to put chunks in.\n bucket_name: \n\n # The size of the buffer that the GCS client uses for each PUT request. 0\n # to disable buffering.\n [chunk_buffer_size: | default = 0]\n\n # The duration after which the requests to GCS should be timed out.\n [request_timeout: | default = 0s]\n\n# Configures storing chunks in Cassandra\ncassandra:\n # Comma-separated hostnames or IPs of Cassandra instances\n addresses: \n\n # Port that cassandra is running on\n [port: | default = 9042]\n\n # Keyspace to use in Cassandra\n keyspace: \n\n # Consistency level for Cassandra\n [consistency: | default = \"QUORUM\"]\n\n # Replication factor to use in Cassandra.\n [replication_factor: | default = 1]\n\n # Instruct the Cassandra driver to not attempt to get host\n # info from the system.peers table.\n [disable_initial_host_lookup: | default = false]\n\n # Use SSL when connecting to Cassandra instances.\n [SSL: | default = false]\n\n # Require SSL certificate validation when SSL is enabled.\n [host_verification: | default = true]\n\n # Path to certificate file to verify the peer when SSL is\n # enabled.\n [CA_path: ]\n\n # Enable password authentication when connecting to Cassandra.\n [auth: | default = false]\n\n # Username for password authentication when auth is true.\n [username: ]\n\n # Password for password authentication when auth is true.\n [password: ]\n\n # Timeout when connecting to Cassandra.\n [timeout: | default = 600ms]\n\n # Initial connection timeout during initial dial to server.\n [connect_timeout: | default = 600ms]\n\n# Configures storing index in BoltDB. Required fields only\n# required when boltdb is present in config.\nboltdb:\n # Location of BoltDB index files.\n directory: \n\n# Configures storing the chunks on the local filesystem. Required\n# fields only required when filesystem is present in config.\nfilesystem:\n # Directory to store chunks in.\n directory: \n\n# Cache validity for active index entries. Should be no higher than\n# the chunk_idle_period in the ingester settings.\n[indexcachevalidity: | default = 5m]\n\n# The maximum number of chunks to fetch per batch.\n[max_chunk_batch_size: | default = 50]\n\n# Config for how the cache for index queries should\n# be built.\nindex_queries_cache_config: \n```\n\n### cache_config\n\nThe `cache_config` block configures how Loki will cache requests, chunks, and\nthe index to a backing cache store.\n\n```yaml\n# Enable in-memory cache.\n[enable_fifocache: ]\n\n# The default validity of entries for caches unless overridden.\n# \"defaul\" is correct.\n[defaul_validity: ]\n\n# Configures the background cache when memcached is used.\nbackground:\n # How many goroutines to use to write back to memcached.\n [writeback_goroutines: | default = 10]\n\n # How many chunks to buffer for background write back to memcached.\n [writeback_buffer: = 10000]\n\n# Configures memcached settings.\nmemcached:\n # Configures how long keys stay in memcached.\n expiration: \n\n # Configures how many keys to fetch in each batch request.\n batch_size: \n\n # Maximum active requests to memcached.\n [parallelism: | default = 100]\n\n# Configures how to connect to one or more memcached servers.\nmemcached_client:\n # The hostname to use for memcached services when caching chunks. If\n # empty, no memcached will be used. A SRV lookup will be used.\n [host: ]\n\n # SRV service used to discover memcached servers.\n [service: | default = \"memcached\"]\n\n # Maximum time to wait before giving up on memcached requests.\n [timeout: | default = 100ms]\n\n # The maximum number of idle connections in the memcached client\n # pool.\n [max_idle_conns: | default = 100]\n\n # The period with which to poll the DNS for memcached servers.\n [update_interval: | default = 1m]\n\n # Whether or not to use a consistent hash to discover multiple memcached\n # servers.\n [consistent_hash: ]\n\nredis:\n # Redis service endpoint to use when caching chunks. If empty, no redis will be used.\n [endpoint: ]\n # Maximum time to wait before giving up on redis requests.\n [timeout: | default = 100ms]\n # How long keys stay in the redis.\n [expiration: | default = 0s]\n # Maximum number of idle connections in pool.\n [max_idle_conns: | default = 80]\n # Maximum number of active connections in pool.\n [max_active_conns: | default = 0]\n\nfifocache:\n # Number of entries to cache in-memory.\n [size: | default = 0]\n\n # The expiry duration for the in-memory cache.\n [validity: | default = 0s]\n```\n\n## chunk_store_config\n\nThe `chunk_store_config` block configures how chunks will be cached and how long\nto wait before saving them to the backing store.\n\n```yaml\n# The cache configuration for storing chunks\n[chunk_cache_config: ]\n\n# The cache configuration for deduplicating writes\n[write_dedupe_cache_config: ]\n\n# The minimum time between a chunk update and being saved\n# to the store.\n[min_chunk_age: ]\n\n# Cache index entries older than this period. Default is\n# disabled.\n[cache_lookups_older_than: ]\n\n# Limit how long back data can be queried. Default is disabled.\n# This should always be set to a value less than or equal to\n# what is set in `table_manager.retention_period`.\n[max_look_back_period: ]\n```\n\n## schema_config\n\nThe `schema_config` block configures schemas from given dates.\n\n```yaml\n# The configuration for chunk index schemas.\nconfigs:\n - []\n```\n\n### period_config\n\nThe `period_config` block configures what index schemas should be used\nfor from specific time periods.\n\n```yaml\n# The date of the first day that index buckets should be created. Use\n# a date in the past if this is your only period_config, otherwise\n# use a date when you want the schema to switch over.\n[from: ]\n\n# store and object_store below affect which key is\n# used.\n\n# Which store to use for the index. Either cassandra, bigtable, dynamodb, or\n# boltdb\nstore: \n\n# Which store to use for the chunks. Either gcs, s3, inmemory, filesystem,\n# cassandra. If omitted, defaults to same value as store.\n[object_store: ]\n\n# The schema to use. Set to v9 or v10.\nschema: \n\n# Configures how the index is updated and stored.\nindex:\n # Table prefix for all period tables.\n prefix: \n # Table period.\n [period: | default = 168h]\n # A map to be added to all managed tables.\n tags:\n [: ...]\n\n# Configured how the chunks are updated and stored.\nchunks:\n # Table prefix for all period tables.\n prefix: \n # Table period.\n [period: | default = 168h]\n # A map to be added to all managed tables.\n tags:\n [: ...]\n\n# How many shards will be created. Only used if schema is v10.\n[row_shards: | default = 16]\n```\n\nWhere `daytime` is a value in the format of `yyyy-mm-dd` like `2006-01-02`.\n\n## limits_config\n\nThe `limits_config` block configures global and per-tenant limits for ingesting\nlogs in Loki.\n\n```yaml\n# Whether the ingestion rate limit should be applied individually to each\n# distributor instance (local), or evenly shared across the cluster (global).\n# The ingestion rate strategy cannot be overridden on a per-tenant basis.\n#\n# - local: enforces the limit on a per distributor basis. The actual effective\n# rate limit will be N times higher, where N is the number of distributor\n# replicas.\n# - global: enforces the limit globally, configuring a per-distributor local\n# rate limiter as \"ingestion_rate / N\", where N is the number of distributor\n# replicas (it's automatically adjusted if the number of replicas change).\n# The global strategy requires the distributors to form their own ring, which\n# is used to keep track of the current number of healthy distributor replicas.\n[ingestion_rate_strategy: | default = \"local\"]\n\n# Per-user ingestion rate limit in sample size per second. Units in MB.\n[ingestion_rate_mb: | default = 4]\n\n# Per-user allowed ingestion burst size (in sample size). Units in MB.\n# The burst size refers to the per-distributor local rate limiter even in the\n# case of the \"global\" strategy, and should be set at least to the maximum logs\n# size expected in a single push request.\n[ingestion_burst_size_mb: | default = 6]\n\n# Maximum length of a label name.\n[max_label_name_length: | default = 1024]\n\n# Maximum length of a label value.\n[max_label_value_length: | default = 2048]\n\n# Maximum number of label names per series.\n[max_label_names_per_series: | default = 30]\n\n# Whether or not old samples will be rejected.\n[reject_old_samples: | default = false]\n\n# Maximum accepted sample age before rejecting.\n[reject_old_samples_max_age: | default = 336h]\n\n# Duration for a table to be created/deleted before/after it's\n# needed. Samples won't be accepted before this time.\n[creation_grace_period: | default = 10m]\n\n# Enforce every sample has a metric name.\n[enforce_metric_name: | default = true]\n\n# Maximum number of active streams per user, per ingester. 0 to disable.\n[max_streams_per_user: | default = 10000]\n\n# Maximum number of active streams per user, across the cluster. 0 to disable.\n# When the global limit is enabled, each ingester is configured with a dynamic\n# local limit based on the replication factor and the current number of healthy\n# ingesters, and is kept updated whenever the number of ingesters change.\n[max_global_streams_per_user: | default = 0]\n\n# Maximum number of chunks that can be fetched by a single query.\n[max_chunks_per_query: | default = 2000000]\n\n# The limit to length of chunk store queries. 0 to disable.\n[max_query_length: | default = 0]\n\n# Maximum number of queries that will be scheduled in parallel by the\n# frontend.\n[max_query_parallelism: | default = 14]\n\n# Cardinality limit for index queries\n[cardinality_limit: | default = 100000]\n\n# Maximum number of stream matchers per query.\n[max_streams_matchers_per_query: | default = 1000]\n\n# Filename of per-user overrides file\n[per_tenant_override_config: ]\n\n# Period with which to reload the overrides file if configured.\n[per_tenant_override_period: | default = 10s]\n```\n\n## table_manager_config\n\nThe `table_manager_config` block configures how the table manager operates\nand how to provision tables when DynamoDB is used as the backing store.\n\n```yaml\n# Master 'off-switch' for table capacity updates, e.g. when troubleshooting\n[throughput_updates_disabled: | default = false]\n\n# Master 'on-switch' for table retention deletions\n[retention_deletes_enabled: | default = false]\n\n# How far back tables will be kept before they are deleted. 0s disables\n# deletion. The retention period must be a multiple of the index / chunks\n# table \"period\" (see period_config).\n[retention_period: | default = 0s]\n\n# Period with which the table manager will poll for tables.\n[dynamodb_poll_interval: | default = 2m]\n\n# duration a table will be created before it is needed.\n[creation_grace_period: | default = 10m]\n\n# Configures management of the index tables for DynamoDB.\nindex_tables_provisioning: \n\n# Configures management of the chunk tables for DynamoDB.\nchunk_tables_provisioning: \n```\n\n### provision_config\n\nThe `provision_config` block configures provisioning capacity for DynamoDB.\n\n```yaml\n# Enables on-demand throughput provisioning for the storage\n# provider, if supported. Applies only to tables which are not autoscaled.\n[provisioned_throughput_on_demand_mode: | default = false]\n\n# DynamoDB table default write throughput.\n[provisioned_write_throughput: | default = 3000]\n\n# DynamoDB table default read throughput.\n[provisioned_read_throughput: | default = 300]\n\n# Enables on-demand throughput provisioning for the storage provide,\n# if supported. Applies only to tables which are not autoscaled.\n[inactive_throughput_on_demand_mode: | default = false]\n\n# DynamoDB table write throughput for inactive tables.\n[inactive_write_throughput: | default = 1]\n\n# DynamoDB table read throughput for inactive tables.\n[inactive_read_throughput: | Default = 300]\n\n# Active table write autoscale config.\n[write_scale: ]\n\n# Inactive table write autoscale config.\n[inactive_write_scale: ]\n\n# Number of last inactive tables to enable write autoscale.\n[inactive_write_scale_lastn: ]\n\n# Active table read autoscale config.\n[read_scale: ]\n\n# Inactive table read autoscale config.\n[inactive_read_scale: ]\n\n# Number of last inactive tables to enable read autoscale.\n[inactive_read_scale_lastn: ]\n```\n\n#### auto_scaling_config\n\nThe `auto_scaling_config` block configures autoscaling for DynamoDB.\n\n```yaml\n# Whether or not autoscaling should be enabled.\n[enabled: : default = false]\n\n# AWS AutoScaling role ARN\n[role_arn: ]\n\n# DynamoDB minimum provision capacity.\n[min_capacity: | default = 3000]\n\n# DynamoDB maximum provision capacity.\n[max_capacity: | default = 6000]\n\n# DynamoDB minimum seconds between each autoscale up.\n[out_cooldown: | default = 1800]\n\n# DynamoDB minimum seconds between each autoscale down.\n[in_cooldown: | default = 1800]\n\n# DynamoDB target ratio of consumed capacity to provisioned capacity.\n[target: | default = 80]\n```\n", "content_chars": 29804, "published_date": null }, { "rank": 8, "url": "https://grafana.com/docs/enterprise-logs/latest/manage/storage/retention/", "title": "Log retention | Grafana Enterprise Logs documentation", "content": "Log retention | Grafana Enterprise Logs documentation\n\nSave the Date!\n\nSign up to get the latest GrafanaCON 2026 updates, agenda drops, and first access to early-bird tickets when they\u2019re first available.\n\nWe have our sightson Barcelona\n\nSave the Date\n\nWatch the keynote again and signup for more on-demand sessions.\n\n\ud83d\udce2 Registration + agenda now live Explore the latest Grafana Cloud and AI solutions, learn tips & tricks from demos and hands-on workshops, and get actionable advice to advance your observability strategy. Register now and get 50% off - limited tickets available (while they last!).\n\nLog retention\n\nEnterprise\n\n# Log retention\n\nRetention in Grafana Loki is achieved through the Compactor. By default the`compactor.retention-enabled` flag is not set, so the logs sent to Loki live forever.\n\nNote\n\nIf you have a lifecycle policy configured on the object store, please ensure that it is longer than the retention period.\n\nGranular retention policies to apply retention at per tenant or per stream level are also supported by the Compactor.\n\nNote\n\nThe Compactor does not support retention on legacy index types. Please use the Table Manager when using legacy index types. Both the Table manager and legacy index types are deprecated and may be removed in future major versions of Loki.\n\n## Compactor\n\nThe Compactor is responsible for compaction of index files and applying log retention.\n\nNote\n\nRun the Compactor as a singleton (a single instance).\n\nThe Compactor loops to apply compaction and retention at every`compactor.compaction-interval`, or as soon as possible if running behind. Both compaction and retention are idempotent, which means once the action has been performed, if the action is performed multiple times, it has no further effect on logs after the first time it is performed. If the Compactor restarts, it continues from where it left off.\n\nNote\n\nChanges to your retention period are not retroactive, that is, they are not applied to logs that have already been ingested.\n\nThe Compactor\u2019s algorithm to apply retention is as follows:\n\nFor each day or table (one table per day with 24h index period):\n\n- Compact multiple index files in the table into per-tenant index files. Result of compaction is a single index file per tenant per day.\n- Traverse the per-tenant index. Use the tenant configuration to identify the chunks that need to be removed.\n- Remove the references to the matching chunks from the index and add the chunk references to a marker file on disk.\n- Upload the new modified index files.\n\nChunks are not deleted while applying the retention algorithm on the index. They are deleted asynchronously by a sweeper process and this delay can be configured by setting`-compactor.retention-delete-delay`. Marker files are used to keep track of the chunks pending for deletion.\n\nChunks cannot be deleted immediately for the following reasons:\n\nIndex Gateway downloads a copy of the index files to serve queries and refreshes them at a regular interval. Having a delay allows the index gateways to pull the modified index file which would not contain any reference to the chunks marked for deletion. Without the delay, index files (that are stale) on the gateways could refer to already deleted chunks leading to query failures.\n\nIt provides a short window of time in which to cancel chunk deletion in the case of a configuration mistake.\n\nMarker files should be stored on a persistent disk to ensure that the chunks pending for deletion are processed even if the Compactor process restarts.\n\nNote\n\nGrafana Labs recommends running Compactor as a stateful deployment (StatefulSet when using Kubernetes) with a persistent storage for storing marker files.\n\n### Retention Configuration\n\nThis Compactor configuration example activates retention.\n\nYAML Copy\n\nExpand code\n\n```yaml\ncompactor:\n working_directory: /data/retention\n compaction_interval: 10m\n retention_enabled: true\n retention_delete_delay: 2h\n retention_delete_worker_count: 150\n delete_request_store: gcs\nschema_config:\n configs:\n - from: \"2020-07-31\"\n index:\n period: 24h\n prefix: index_\n object_store: gcs\n schema: v13\n store: tsdb\nstorage_config:\n tsdb_shipper:\n active_index_directory: /data/index\n cache_location: /data/index_cache\n gcs:\n bucket_name: loki\n```\n\nNote\n\nRetention is only available if the index period is 24h. Single store TSDB and single store BoltDB require 24h index period.\n\n`retention_enabled` should be set to true. Without this, the Compactor will only compact tables.\n\n`delete_request_store` should be set to configure the store for delete requests. This is required when retention is enabled.\n\n`working_directory` is the directory where marked chunks and temporary tables will be saved.\n\n`compaction_interval` dictates how often compaction and/or retention is applied. If the Compactor falls behind, compaction and/or retention occur as soon as possible.\n\n`retention_delete_delay` is the delay after which the Compactor will delete marked chunks.\n\n`retention_delete_worker_count` specifies the maximum quantity of goroutine workers instantiated to delete chunks.\n\n#### Configuring the retention period\n\nRetention period is configured within the limits_config configuration section.\n\nThere are two ways of setting retention policies:\n\n- `retention_period` which is applied globally for all log streams.\n- `retention_stream` which is only applied to log streams matching the selector.\n\nNote\n\nThe minimum retention period is 24h.\n\nThis example configures global retention that applies to all tenants (unless overridden by configuring per-tenant overrides):\n\nYAML Copy\n\n```yaml\n...\nlimits_config:\n retention_period: 744h\n retention_stream:\n - selector: '{namespace=\"dev\"}'\n priority: 1\n period: 24h\n per_tenant_override_config: /etc/overrides.yaml\n...\n```\n\nNote\n\nYou can only use label matchers in the`selector` field of a`retention_stream` definition. Arbitrary LogQL expressions are not supported.\n\nPer tenant retention can be defined by configuring runtime overrides. For example:\n\nYAML Copy\n\n```yaml\noverrides:\n \"29\":\n retention_period: 168h\n retention_stream:\n - selector: '{namespace=\"prod\"}'\n priority: 2\n period: 336h\n - selector: '{container=\"loki\"}'\n priority: 1\n period: 72h\n \"30\":\n retention_stream:\n - selector: '{container=\"nginx\", level=\"debug\"}'\n priority: 1\n period: 24h\n```\n\nRetention period for a given stream is decided based on the first match in this list:\n\n1. If multiple per-tenant`retention_stream` selectors match the stream, retention period with the highest priority is picked.\n2. If multiple global`retention_stream` selectors match the stream, retention period with the highest priority is picked. This value is not considered if per-tenant`retention_stream` is set.\n3. If a per-tenant`retention_period` is specified, it will be applied.\n4. The global`retention_period` will be applied if none of the above match.\n5. If no global`retention_period` is specified, the default value of`0s` is used, which means logs are kept indefinitely.\n\nNote\n\nThe larger the priority value, the higher the priority.\n\nStream matching uses the same syntax as Prometheus label matching:\n\n- `=`: Select labels that are exactly equal to the provided string.\n- `!=`: Select labels that are not equal to the provided string.\n- `=~`: Select labels that regex-match the provided string.\n- `!~`: Select labels that do not regex-match the provided string.\n\nThe example configurations defined above will result in the following retention periods:\n\nFor tenant`29`:\n\n- Streams that have the namespace label`prod` will have a retention period of`336h`(2 weeks), even if the container label is`loki`, since the priority of the`prod` rule is higher.\n- Streams that have the container label`loki` but are not in the namespace`prod` will have a`72h` retention period.\n- For the rest of the streams in this tenant, per-tenant override`retention_period` value of`168h` is applied.\n\nFor tenant`30`:\n\n- Streams that have the label`nginx` and level`debug` will have a retention period of`24h`.\n- For the rest of the streams in this tenant the global retention period of`744h`, since there is no override specified.\n\nAll tenants except`29` and`30`:\n\n- Streams that have the namespace label`dev` will have a retention period of`24h` hours.\n- Streams except those with the namespace label`dev` will have the retention period of`744h`.\n\nNote\n\nIf you are a Grafana Cloud customer, you can use the config self-serve API to configure your tenant retention.\n\n## Table Manager (deprecated)\n\nRetention through the Table Manager is achieved by relying on the object store TTL feature, and will work for both boltdb-shipper store and chunk/index stores.\n\nIn order to enable the retention support, the Table Manager needs to be configured to enable deletions and a retention period. Please refer to the table_manager section of the Loki configuration reference for all available options. Alternatively, the`table-manager.retention-period` and`table-manager.retention-deletes-enabled` command line flags can be used. The provided retention period needs to be a duration represented as a string that can be parsed using the Prometheus common model ParseDuration. Examples:`7d`,`1w`,`168h`.\n\nWarning\n\nThe retention period must be a multiple of the index and chunks table`period`, configured in the period_config block. See the Table Manager documentation for more information.\n\nNote\n\nTo avoid querying of data beyond the retention period,`max_query_lookback` config in limits_config must be set to a value less than or equal to what is set in`table_manager.retention_period`.\n\nWhen using S3 or GCS, the bucket storing the chunks needs to have the expiry policy set correctly. For more details check S3\u2019s documentation or GCS\u2019s documentation.\n\nIf you must delete ingested logs, you can delete old chunks in your object store. Note, however, that this only deletes the log content and keeps the label index intact; you will still be able to see related labels but will be unable to retrieve the deleted log content.\n\nFor further details on the Table Manager internals, refer to the Table Manager documentation.\n\nAlternatively, if the BoltDB Shipper is configured for the index store, you can enable Log entry deletion to delete log entries from a specific stream.\n\n## Example Configuration\n\nExample configuration with GCS with a 28 day retention:\n\nYAML Copy\n\nExpand code\n\n```yaml\nschema_config:\n configs:\n - from: 2018-04-15\n store: tsdb\n object_store: gcs\n schema: v13\n index:\n prefix: loki_index_\n period: 24h\n\nstorage_config:\n tsdb_shipper:\n active_index_directory: /loki/index\n cache_location: /loki/index_cache\n gcs:\n bucket_name: GCS_BUCKET_NAME\n\nlimits_config:\n max_query_lookback: 672h # 28 days\n retention_period: 672h # 28 days\n\ncompactor:\n working_directory: /data/retention\n delete_request_store: gcs\n retention_enabled: true\n```\n\n## Object store lifecycle policies\n\nLoki stores chunks and indexes as objects in the configured object store (S3, GCS, Azure Blob Storage, and so on). Chunks are deleted by the Compactor according to your configured retention, but the Compactor only removes chunk objects: it never deletes the objects that hold cluster state, such as the index or the cluster seed.\n\nIf you also configure an object store lifecycle policy on the bucket, scope it carefully. A blanket \u201cdelete everything older than N days\u201d rule will eventually remove objects that Loki must keep, which corrupts the store and causes query failures. The note at the top of this page applies here as well: any lifecycle expiration must be longer than your retention period.\n\nWarning\n\nNever apply a lifecycle rule with an empty prefix (one that targets the whole bucket). Always scope lifecycle rules to the per-tenant chunk prefixes described below.\n\n### Bucket object layout\n\nThe following objects are written to the configured bucket. The exact prefixes depend on your`schema_config` and on whether multi-tenancy is enabled, so confirm them against your own configuration before writing a lifecycle rule.\n\nExpand table\n\n| Object or prefix | Content | Safe to expire with a lifecycle policy? |\n| --- | --- | --- |\n| `/`(for example,`fake/` in single-tenant mode, or your org ID per tenant) | Chunk objects (the log data). | Yes. These are the only objects a lifecycle rule should target. |\n| Index objects (under the index path prefix,`index/` by default, set by`index.path_prefix` in`schema_config`) | TSDB or BoltDB index files, written by ingesters and rewritten by the Compactor. The`index.prefix` value (for example`index_` or`loki_index_`) is the table-name prefix used inside this path, not the path itself, so match a lifecycle rule against the path prefix. | No. Loki manages the index lifecycle internally. Deleting index objects removes references to live chunks and breaks queries. |\n| `loki_cluster_seed.json` | A small file at the bucket root holding the cluster seed used by usage reporting. It is written once and never rotated. | No. A blanket age-based rule will delete it once it ages past the time to live (TTL). |\n| Compactor delete-request store (the`delete_request_store` you configure, such as`markers/` and the delete-request objects) | The Compactor\u2019s record of chunks pending deletion and of in-flight delete requests. | No. Removing these makes the Compactor lose track of pending deletions. |\n| Ruler store objects (if the Ruler is backed by the same bucket) | Recording and alerting rule files. | No. These are configuration, not time-series data. |\n\n### Compactor retention versus object store lifecycle policies\n\nCompactor retention and object store lifecycle policies are two independent mechanisms, and they should not be configured in ways that conflict:\n\n- The Compactor is index-aware. It only deletes a chunk after it has removed every reference to that chunk from the index, and it waits`compactor.retention-delete-delay`(default`2h`) before the sweeper deletes the marked chunk, so that index gateways pick up the rewritten index first.\n- An object store lifecycle policy is index-unaware. It deletes objects purely by age and prefix, with no knowledge of whether a chunk is still referenced by the index.\n\nFor most deployments, enabling Compactor retention is sufficient and an object store lifecycle policy is not required. If you do add a lifecycle policy as a cost safety net, set its expiration longer than your retention period plus`compactor.retention-delete-delay`, so the Compactor always deletes chunks first and the lifecycle rule only ever catches objects the Compactor has already orphaned.\n\n### Example: Amazon S3 lifecycle policy\n\nThis policy expires only chunk objects under a single tenant prefix. Replace`fake/` with your tenant prefix (use one rule per tenant, or your org ID prefix, when multi-tenancy is enabled). The`Days` value must be larger than your retention period plus the delete delay.\n\nJSON Copy\n\n```json\n{\n \"Rules\": [\n {\n \"ID\": \"loki-chunk-expiration\",\n \"Status\": \"Enabled\",\n \"Filter\": {\n \"Prefix\": \"fake/\"\n },\n \"Expiration\": {\n \"Days\": 395\n }\n }\n ]\n}\n```\n\nApply it with the AWS CLI:\n\nBash Copy\n\n```bash\naws s3api put-bucket-lifecycle-configuration \\\n --bucket YOUR_LOKI_BUCKET \\\n --lifecycle-configuration file://lifecycle.json\n```\n\n### Example: Google Cloud Storage lifecycle rule\n\nThe equivalent GCS rule expires objects by age and prefix. As with S3, scope`matchesPrefix` to the tenant chunk prefix and keep`age` larger than your retention period plus the delete delay.\n\nJSON Copy\n\n```json\n{\n \"rule\": [\n {\n \"action\": {\"type\": \"Delete\"},\n \"condition\": {\n \"age\": 395,\n \"matchesPrefix\": [\"fake/\"]\n }\n }\n ]\n}\n```\n\nApply it with the Google Cloud CLI:\n\nBash Copy\n\n```bash\ngcloud storage buckets update gs://YOUR_LOKI_BUCKET --lifecycle-file=lifecycle.json\n```\n\nFor Azure Blob Storage, use a management policy with a`prefixMatch` set to the tenant chunk prefix and the same \u201cexpiration must exceed retention plus delete delay\u201d rule.\n\n## Was this page helpful?\n\nYes No\n\nHelp and support\n\n## Related resources from Grafana Labs\n\nAdditional helpful documentation, links, and articles:\n\nVideoScaling and securing your logs with Grafana LokiThis webinar covers the challenges of scaling and securing logs, and how Grafana Cloud Logs powered by Grafana Loki can help, cost-effectively.\n\nVideoManaging privacy in log data with Grafana LokiLaws for data privacy are complex and rapidly changing. During this webinar, we will show how Grafana Loki can help organizations meet these requirements.\n\nBlogThe concise guide to Grafana Loki: Everything you need to know about labelsIn Part 2 of \"The concise guide to Loki,\" you'll learn about how to properly use labels in our favorite logging database.\n\nAsk Grafana Assistant", "content_chars": 17004, "published_date": null }, { "rank": 9, "url": "https://community.grafana.com/t/loki-retention-isnt-working-as-expected-logs-older-than-24-hours-are-not-getting-deleted/142440/3", "title": "Loki retention isn't working as expected\u2014logs older than 24 hours are not getting deleted - Grafana Loki - Grafana Labs Community Forums", "content": "Loki retention isn't working as expected\u2014logs older than 24 hours are not getting deleted - Grafana Loki - Grafana Labs Community Forums\n\n# Loki retention isn\u2019t working as expected\u2014logs older than 24 hours are not getting deleted \n\nCheck Grafana events next to you this September From Austin to London to Kenya to Nagpur - 27 locations, 27 opportunities to meet the community \n\n## post by sivaprasad298 on Feb 4, 2025 \n\n 1 Feb 2025 \n\nHi, I have below configuration for grafana loki . I am expecting that the chunks will be deleted after every 24hrs. But it is not working as expected.\n\nauth_enabled: false\n\nserver: http_listen_port: 3100 http_server_read_timeout: 600s http_server_write_timeout: 600s grpc_server_max_recv_msg_size: 18388608 grpc_server_max_send_msg_size: 18388608\n\ncommon: path_prefix: /loki storage: filesystem: chunks_directory: /loki/chunks rules_directory: /loki/rules replication_factor: 1 ring: kvstore: store: inmemory\n\nschema_config: configs: - from: 2020-10-24 store: tsdb object_store: filesystem schema: v11 index: prefix: index_ period: 24h\n\nstorage_config: tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h\n\nlimits_config: enforce_metric_name: false reject_old_samples: true reject_old_samples_max_age: 24h ingestion_rate_mb: 50 ingestion_burst_size_mb: 100 max_global_streams_per_user: 5000 per_stream_rate_limit: 50MB retention_period: 24h\n\ncompactor: retention_enabled: true shared_store: filesystem working_directory: /loki/compactor-work\n\nchunk_store_config: max_look_back_period: 0s\n\ntable_manager: retention_deletes_enabled: true retention_period: 24h\n\ningester_client: remote_timeout: 15s\n\n## post by tonyswumac on Feb 4, 2025 \n\nTry disabling table manager (see Log retention | Grafana Loki documentation).\n\n## post by sivaprasad298 on Feb 5, 2025 \n\n 1 Feb 2025 \n\nThanks for the response, tried that but still not working\n\nThis is my updated configuration:\n\nauth_enabled: false\n\nserver: http_listen_port: 3100 http_server_read_timeout: 600s http_server_write_timeout: 600s grpc_server_max_recv_msg_size: 18388608 grpc_server_max_send_msg_size: 18388608\n\ncommon: path_prefix: /loki storage: filesystem: chunks_directory: /loki/chunks rules_directory: /loki/rules replication_factor: 1 ring: kvstore: store: inmemory\n\nschema_config: configs: - from: 2020-10-24 store: tsdb object_store: filesystem schema: v11 index: prefix: index_ period: 24h\n\nstorage_config: tsdb_shipper: active_index_directory: /loki/index cache_location: /loki/index_cache cache_ttl: 24h\n\nlimits_config: enforce_metric_name: false reject_old_samples: true reject_old_samples_max_age: 24h ingestion_rate_mb: 50 ingestion_burst_size_mb: 100 max_global_streams_per_user: 5000 per_stream_rate_limit: 50MB\n\ncompactor: working_directory: /loki/compactor-work compaction_interval: 10m retention_enabled: true retention_delete_delay: 1h shared_store: filesystem retention_delete_worker_count: 5\n\ningester_client: remote_timeout: 15s\n\n## post by tonyswumac on Feb 5, 2025 \n\n Feb 2025 \n\n1. You are missing `retention_period` on your latest config.\n2. You may want to set `retention_period` to 48h. The minimum index period for Loki is 24h, and there is a chance compactor won\u2019t delete anything until 48h is up. I don\u2019t set my retention this low, so I am not entirely sure what the behavior is here.\n\n## Closed on Feb 5 \n\nThis topic was automatically closed 365 days after the last reply. New replies are no longer allowed.", "content_chars": 3469, "published_date": "2025-02-04T00:00:00.000Z" }, { "rank": 10, "url": "https://github.com/grafana/loki/blob/06ce74b7/docs/sources/shared/configuration.md", "title": "docs/sources/shared/configuration.md", "content": "# docs/sources/shared/configuration.md\n\n- Branch: 06ce74b7\n- Repository: grafana/loki\n\n---\n\n---\ndescription: Configuration reference for the parameters used to configure Grafana Loki.\ntitle: Grafana Loki configuration parameters\n---\n\n \n\n## Printing Loki config at runtime\n\nIf you pass Loki the flag `-print-config-stderr` or `-log-config-reverse-order`, (or `-print-config-stderr=true`)\nLoki will dump the entire config object it has created from the built-in defaults combined first with\noverrides from config file, and second by overrides from flags.\n\nThe result is the value for every config object in the Loki config struct, which is very large...\n\nMany values will not be relevant to your install such as storage configs which you are not using and which you did not define,\nthis is expected as every option has a default value if it is being used or not.\n\nThis config is what Loki will use to run, it can be invaluable for debugging issues related to configuration and\nis especially useful in making sure your config files and flags are being read and loaded properly.\n\n`-print-config-stderr` is nice when running Loki directly e.g. `./loki ` as you can get a quick output of the entire Loki config.\n\n`-log-config-reverse-order` is the flag we run Loki with in all our environments, the config entries are reversed so\nthat the order of configs reads correctly top to bottom when viewed in Grafana's Explore.\n\n## Configuration file reference\n\nTo specify which configuration file to load, pass the `-config.file` flag at the\ncommand line. The value can be a list of comma separated paths, then the first\nfile that exists will be used.\nIf no `-config.file` argument is specified, Loki will look up the `config.yaml` in the\ncurrent working directory and the `config/` subdirectory and try to use that.\n\nThe file is written in YAML\nformat, defined by the scheme below.\nBrackets indicate that a parameter is optional. For non-list parameters the\nvalue is set to the specified default.\n\n### Use environment variables in the configuration\n\n> **Note:** This feature is only available in Loki 2.1+.\n\nYou can use environment variable references in the configuration file to set values that need to be configurable during deployment.\nTo do this, pass `-config.expand-env=true` and use:\n\n```bash\n${VAR}\n```\n\nWhere VAR is the name of the environment variable.\n\nEach variable reference is replaced at startup by the value of the environment variable.\nThe replacement is case-sensitive and occurs before the YAML file is parsed.\nReferences to undefined variables are replaced by empty strings unless you specify a default value or custom error text.\n\nTo specify a default value, use:\n\n```bash\n${VAR:-default_value}\n```\n\nWhere default_value is the value to use if the environment variable is undefined.\n\nPass the `-config.expand-env` flag at the command line to enable this way of setting configs.\n\n### Generic placeholders\n\n- ` ` : a boolean that can take the values `true` or `false`\n- ` ` : any integer matching the regular expression `[1-9]+[0-9]*`\n- ` ` : a duration matching the regular expression `[0-9]+(ns|us|\u00b5s|ms|[smh])`\n- ` ` : a string matching the regular expression `[a-zA-Z_][a-zA-Z0-9_]*`\n- ` ` : a string of unicode characters\n- ` ` : a valid path relative to current working directory or an absolute path.\n- ` ` : a valid string consisting of a hostname or IP followed by an optional port number\n- ` ` : a string\n- ` ` : a string that represents a secret, such as a password\n\n### Supported contents and default values of `loki.yaml`\n\n \n\n```yaml\n# A comma-separated list of components to run. The default value 'all' runs Loki\n# in single binary mode. The value 'read' is an alias to run only read-path\n# related components such as the querier and query-frontend, but all in the same\n# process. The value 'write' is an alias to run only write-path related\n# components such as the distributor and compactor, but all in the same process.\n# Supported values: all, compactor, distributor, ingester, querier,\n# query-scheduler, ingester-querier, query-frontend, index-gateway, ruler,\n# table-manager, read, write. A full list of available targets can be printed\n# when running Loki with the '-list-targets' command line flag.\n# CLI flag: -target\n[target: | default = \"all\"]\n\n# Enables authentication through the X-Scope-OrgID header, which must be present\n# if true. If false, the OrgID will always be set to 'fake'.\n# CLI flag: -auth.enabled\n[auth_enabled: | default = true]\n\n# The amount of virtual memory in bytes to reserve as ballast in order to\n# optimize garbage collection. Larger ballasts result in fewer garbage\n# collection passes, reducing CPU overhead at the cost of heap size. The ballast\n# will not consume physical memory, because it is never read from. It will,\n# however, distort metrics, because it is counted as live memory.\n# CLI flag: -config.ballast-bytes\n[ballast_bytes: | default = 0]\n\n# Configures the server of the launched module(s).\n[server: ]\n\nui:\n # Enable the experimental Loki UI.\n # CLI flag: -ui.enabled\n [enabled: | default = false]\n\n # Enable debug logging for the UI.\n # CLI flag: -ui.debug\n [debug: | default = false]\n\n goldfish:\n # Enable the Goldfish query comparison feature.\n # CLI flag: -ui.goldfish.enable\n [enable: | default = false]\n\n storage:\n # Storage backend type (cloudsql, rds, or empty for no storage)\n # CLI flag: -ui.goldfish.storage.type\n [type: | default = \"\"]\n\n # CloudSQL host for Goldfish database.\n # CLI flag: -ui.goldfish.storage.cloudsql.host\n [cloudsql_host: | default = \"127.0.0.1\"]\n\n # CloudSQL port for Goldfish database.\n # CLI flag: -ui.goldfish.storage.cloudsql.port\n [cloudsql_port: | default = 3306]\n\n # CloudSQL database name for Goldfish.\n # CLI flag: -ui.goldfish.storage.cloudsql.database\n [cloudsql_database: | default = \"goldfish\"]\n\n # CloudSQL username for Goldfish database.\n # CLI flag: -ui.goldfish.storage.cloudsql.user\n [cloudsql_user: | default = \"\"]\n\n # RDS endpoint (host:port)\n # CLI flag: -ui.goldfish.storage.rds.endpoint\n [rds_endpoint: | default = \"\"]\n\n # RDS database name\n # CLI flag: -ui.goldfish.storage.rds.database\n [rds_database: | default = \"\"]\n\n # RDS database user\n # CLI flag: -ui.goldfish.storage.rds.user\n [rds_user: | default = \"\"]\n\n # Maximum number of database connections for Goldfish.\n # CLI flag: -ui.goldfish.max-connections\n [max_connections: | default = 10]\n\n # Maximum idle time for database connections in seconds.\n # CLI flag: -ui.goldfish.max-idle-time\n [max_idle_time_seconds: | default = 300]\n\n # Base URL of Grafana instance for explore links.\n # CLI flag: -ui.goldfish.grafana-url\n [grafana_url: | default = \"\"]\n\n # UID of the traces datasource in Grafana.\n # CLI flag: -ui.goldfish.traces-datasource-uid\n [traces_datasource_uid: | default = \"\"]\n\n # UID of the Loki datasource in Grafana.\n # CLI flag: -ui.goldfish.logs-datasource-uid\n [logs_datasource_uid: | default = \"\"]\n\n # Namespace for Cell A logs.\n # CLI flag: -ui.goldfish.cell-a-namespace\n [cell_a_namespace: | default = \"\"]\n\n # Namespace for Cell B logs.\n # CLI flag: -ui.goldfish.cell-b-namespace\n [cell_b_namespace: | default = \"\"]\n\n # Results storage backend (gcs, s3) for fetching stored query results.\n # CLI flag: -ui.goldfish.results-backend\n [results_backend: | default = \"\"]\n\n # The thanos_object_store_config block configures the connection to object\n # storage backend using thanos-io/objstore clients. This will become the\n # default way of configuring object store clients in future releases.\n # Currently this is opt-in and takes effect only when `-use-thanos-objstore`\n # is set to true.\n # The CLI flags prefix for this block configuration is: ui.goldfish.results\n [results_bucket: ]\n\n ring:\n kvstore:\n # Backend storage to use for the ring. Supported values are: consul, etcd,\n # inmemory, memberlist, multi.\n # CLI flag: -ui.ring.store\n [store: | default = \"consul\"]\n\n # The prefix for the keys in the store. Should end with a /.\n # CLI flag: -ui.ring.prefix\n [prefix: | default = \"collectors/\"]\n\n # Configuration for a Consul client. Only applies if the selected kvstore\n # is consul.\n # The CLI flags prefix for this block configuration is: ui.ring\n [consul: ]\n\n # Configuration for an ETCD v3 client. Only applies if the selected\n # kvstore is etcd.\n # The CLI flags prefix for this block configuration is: ui.ring\n [etcd: ]\n\n multi:\n # Primary backend storage used by multi-client.\n # CLI flag: -ui.ring.multi.primary\n [primary: | default = \"\"]\n\n # Secondary backend storage used by multi-client.\n # CLI flag: -ui.ring.multi.secondary\n [secondary: | default = \"\"]\n\n # Mirror writes to the secondary store.\n # CLI flag: -ui.ring.multi.mirror-enabled\n [mirror_enabled: | default = false]\n\n # Timeout for storing a value to the secondary store.\n # CLI flag: -ui.ring.multi.mirror-timeout\n [mirror_timeout: | default = 2s]\n\n # Period at which to heartbeat to the ring.\n # CLI flag: -ui.ring.heartbeat-period\n [heartbeat_period: | default = 15s]\n\n # The heartbeat timeout after which compactors are considered unhealthy\n # within the ring. 0 = never (timeout disabled).\n # CLI flag: -ui.ring.heartbeat-timeout\n [heartbeat_timeout: | default = 1m]\n\n # File path where tokens are stored. If empty, tokens are not stored at\n # shutdown and restored at startup.\n # CLI flag: -ui.ring.tokens-file-path\n [tokens_file_path: | default = \"\"]\n\n # True to enable zone-awareness and replicate blocks across different\n # availability zones.\n # CLI flag: -ui.ring.zone-awareness-enabled\n [zone_awareness_enabled: | default = false]\n\n # Number of tokens to own in the ring.\n # CLI flag: -ui.ring.num-tokens\n [num_tokens: | default = 128]\n\n # Factor for data replication.\n # CLI flag: -ui.ring.replication-factor\n [replication_factor: | default = 3]\n\n # Instance ID to register in the ring.\n # CLI flag: -ui.ring.instance-id\n [instance_id: | default = \"\"]\n\n # Name of network interface to read address from.\n # CLI flag: -ui.ring.instance-interface-names\n [instance_interface_names: | default = []]\n\n # Port to advertise in the ring (defaults to server.grpc-listen-port).\n # CLI flag: -ui.ring.instance-port\n [instance_port: | default = 0]\n\n # IP address to advertise in the ring.\n # CLI flag: -ui.ring.instance-addr\n [instance_addr: | default = \"\"]\n\n # The availability zone where this instance is running. Required if\n # zone-awareness is enabled.\n # CLI flag: -ui.ring.instance-availability-zone\n [instance_availability_zone: | default = \"\"]\n\n # Enable using a IPv6 instance address.\n # CLI flag: -ui.ring.instance-enable-ipv6\n [instance_enable_ipv6: | default = false]\n\n# Configures the distributor.\n[distributor: ]\n\n# Configures the querier. Only appropriate when running all modules or just the\n# querier.\n[querier: ]\n\nquery_engine:\n # Experimental: Enable next generation query engine for supported queries.\n # CLI flag: -query-engine.enable\n [enable: | default = false]\n\n # Experimental: Enable distributed query execution.\n # CLI flag: -query-engine.distributed\n [distributed: | default = false]\n\n # Experimental: Name of network interface to read an advertise address from\n # for accepting incoming traffic from query-engine-worker instances when\n # distributed execution is enabled.\n # CLI flag: -query-engine.instance-interface-names\n [instance_interface_names: | default = []]\n\n # Experimental: Batch size of the next generation query engine.\n # CLI flag: -query-engine.batch-size\n [batch_size: | default = 100]\n\n # Experimental: The number of inputs that are prefetched simultaneously by any\n # Merge node. A value of 0 means that only the currently processed input is\n # prefetched, 1 means that only the next input is prefetched, and so on. A\n # negative value means that all inputs are be prefetched in parallel.\n # CLI flag: -query-engine.merge-prefetch-count\n [merge_prefetch_count: | default = 0]\n\n # Configures how to read byte ranges from object storage when using the V2\n # engine.\n range_reads:\n # Experimental: maximum number of parallel reads\n # CLI flag: -query-engine.range-reads.max-parallelism\n [max_parallelism: | default = 10]\n\n # Experimental: maximum distance (in bytes) between ranges that causes them\n # to be coalesced into a single range\n # CLI flag: -query-engine.range-reads.coalesce-size\n [coalesce_size: | default = 1048576]\n\n # Experimental: maximum size of a byte range\n # CLI flag: -query-engine.range-reads.max-range-size\n [max_range_size: | default = 8388608]\n\n # Experimental: minimum size of a byte range\n # CLI flag: -query-engine.range-reads.min-range-size\n [min_range_size: | default = 1048576]\n\n # Experimental: Number of worker threads to spawn. Each worker thread runs one\n # task at a time. 0 means to use GOMAXPROCS value.\n # CLI flag: -query-engine.worker-threads\n [worker_threads: | default = 0]\n\n # Experimental: Address holding DNS SRV records of schedulers to connect to.\n # CLI flag: -query-engine.scheduler-lookup-address\n [scheduler_lookup_address: | default = \"\"]\n\n # Experimental: Interval at which to lookup new schedulers by DNS SRV records.\n # CLI flag: -query-engine.scheduler-lookup-interval\n [scheduler_lookup_interval: | default = 10s]\n\n # Amount of time until data objects are available.\n # CLI flag: -query-engine.storage-lag\n [storage_lag: | default = 1h]\n\n # Initial date when data objects became available. Format YYYY-MM-DD. If not\n # set, assume data objects are always available no matter how far back.\n # CLI flag: -query-engine.storage-start-date\n [storage_start_date: