Kinro documentation

Quickstart

From nothing to a daily digest, without writing code. You need Docker and read access to your logs: a folder of log files, a Loki or a Graylog. The examples use three services, edge/api, core/indexer and notify/mailer, and a job id that passes between them.

1. Get the image

   The engine ships as a Docker image; nothing else is installed on the host. The image is private for now: log in with the pull token you were given.

   $ docker login -u <user>
   $ docker pull kybalda/utyg:0.1.3

2. See what is in your logs

   discover only reads. It writes discovery.json, report.html and index.sqlite into the output folder and prints a summary. Pick where your logs are:

   Log files Loki Graylog

   $ mkdir -p discovery
   $ docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/logs:/logs:ro" -v "$PWD/discovery:/out" \
       kybalda/utyg:0.1.3 discover --logs /logs --out /out

   The summary looks like this. Every secret and every piece of personal data is masked:

   kinro discover: 3 services, 1214 records, 0.1 MB from 3 files in 0.1 s
   services (tier, records, errors, entity ids):
     L1 core/indexer: 559 records, 84 errors; tenant
     L1 edge/api: 341 records, 0 errors; job_ref, tenant_id
     L0 notify/mailer: 314 records, 13 errors; job
   keys across services: 2 (2 confirmed links, 1 unconfirmed candidates)
     k1 job: edge/api:job_ref, notify/mailer:job
     k2 tenant: core/indexer:tenant, edge/api:tenant_id
   flows: 2
     job (cross-service): 300 entities, 4 transitions
     tenant (cross-service): 40 entities, 2 transitions
   findings: 1 error bursts, 1 repeated failures, 4 top error templates, 1 silent gaps
     core/indexer: 80 errors in 2 hot minutes (peak 43/min; usually 1 in an error minute, 0.01/min on average) [2026-10-16T10:30:00Z .. 2026-10-16T10:31:59Z]
       connection refused
     notify/mailer: 1 job values hit the same failure 5+ times (max 12) [2026-10-16T12:52:36Z .. 2026-10-16T13:03:36Z]
       job <hex> failed: upstream unavailable
   secrets: 2, personal data: 2 (values masked)
     edge/api: credential `access_token` ****y0Ge in 2 records
     notify/mailer: bearer_token `bearer_token` ****2Nzg in 1 records
   timeline index: 1957 events of 1410 id values in 889 records, 0.2 MB, 0.0 s; caps hit: none
   wrote /out/discovery.json, /out/report.html and /out/index.sqlite

   Open discovery/report.html in a browser for the full picture: templates with masked example lines, flows with latencies, every finding, what the next log tier would add per service, and a search box for ids.

3. Follow one id

   Take an id you care about, such as a job id from your logs or a support ticket, and see every event it took part in across services. The report shows ids masked; timeline takes the full value you type:

   $ docker run --rm -v "$PWD/discovery:/out:ro" kybalda/utyg:0.1.3 timeline 9d37ff01fe80 --index /out --limit 4
   kinro timeline 9d37ff01fe80: 14 events in 2 services (edge/api, notify/mailer), 2026-10-16T12:52:34.872Z .. 2026-10-16T13:04:36.488Z
   showing the last 4 of 14 events (--limit)
   key k1 job (confirmed link across services)
   2026-10-16T13:01:36.488Z  notify/mailer  error    job 9d37ff01fe80 failed: upstream unavailable  [job]
   2026-10-16T13:02:36.488Z  notify/mailer  error    job 9d37ff01fe80 failed: upstream unavailable  [job]
   2026-10-16T13:03:36.488Z  notify/mailer  error    job 9d37ff01fe80 failed: upstream unavailable  [job]
   2026-10-16T13:04:36.488Z  notify/mailer  info     job 9d37ff01fe80 delivered  [job]
   related ids in the same records (keys followed one hop)
     26389  edge/api:latency_us  1 record
     b14aed54bb69e1f0  edge/api:trace  1 record

4. Watch without rules

   kinro run without --project reads every service of the sources named in the environment and runs the generic monitors. Put the settings in .env:

   # sources: any of Loki, Graylog and files
   KINRO_LOKI_URL=https://loki.internal:3100
   KINRO_LOKI_TOKEN=...
   # KINRO_GRAYLOG_URL=https://graylog.internal:9000
   # KINRO_GRAYLOG_TOKEN=...
   # KINRO_LOG_GLOB=/logs/*/*.log
   KINRO_STATE_PATH=/state/kinro.db
   KINRO_PAGE=error_spike,silent_service
   # TELEGRAM_BOT_TOKEN=...
   # TELEGRAM_CHAT_ID=...

   Check it offline, then start it with a volume for the state:

   $ docker run --rm --env-file .env kybalda/utyg:0.1.3 check
   zero-config: sources loki; monitors new_error_template, error_spike, silent_service, repeated_failure; learning 24h per service; paging error_spike, silent_service; digest 09:00 UTC; state /state/kinro.db; 0 errors, 0 warnings
   $ docker run -d --name kinro --init --restart unless-stopped --env-file .env -v kinro-state:/state \
       kybalda/utyg:0.1.3 run

   For log files, add -v /var/log/myapp:/logs:ro and set KINRO_LOG_GLOB. Existing files are read from their end unless KINRO_LOG_START=beginning.

5. Read the digest

   Each service is learned for its first 24 hours. After that, what the monitors find is kept as an insight and listed in one digest a day at 09:00 UTC. Only the monitors in KINRO_PAGE are sent at once. Without Telegram or Slack settings, messages go to the log, and to KINRO_ALERTS_PATH when it is set.

   [KINRO DIGEST] 2026-10-17 (09:00 UTC)
   3 services watched (0 still learning), 1215 records, 97 errors
   New insights: 5
     new_error_template (2):
       - core/indexer: new error template "connection refused" first seen 2026-10-16 10:30 UTC
       - notify/mailer: new error template "job <hex> failed: upstream unavailable" first seen 2026-10-16 12:52 UTC
     repeated_failure (1):
       - notify/mailer: job=****fe80 hit "job <hex> failed: upstream unavailable" 5 times in 4m (a retry loop?)
     silent_service (2, 2 paged):
       - edge/api: no records since 2026-10-16 13:58:58 UTC (58m); its longest learned pause is 7m
       - notify/mailer: no records since 2026-10-16 13:59:00 UTC (57m); its longest learned pause is 12m
   Ongoing:
     - silent_service: edge/api silent since 2026-10-16 13:58 UTC
     - silent_service: notify/mailer silent since 2026-10-16 13:59 UTC
     - repeated_failure: notify/mailer "job <hex> failed: upstream unavailable", latest id looping at 2026-10-16 12:56 UTC

   status shows the learning progress and the insight counts at any time, safely next to the running container:

   $ docker exec kinro kinro status
   ...
   zero-config: 3 services (0 learning, 24h each); paging error_spike, silent_service; digest 09:00 UTC, next 2026-10-18T09:00:00+00:00, last 2026-10-17
   monitors: new_error_template, error_spike, silent_service, repeated_failure
   insights: new_error_template=2 (0 paged), repeated_failure=1 (0 paged), silent_service=2 (2 paged)
   learning shadow: none; rate-limited: none
     core/indexer: 100% learned, 559 records, 84 errors, last 2026-10-16T14:57:04.513000+00:00
     edge/api: 100% learned, 341 records, 0 errors, last 2026-10-16T13:58:58.697000+00:00, silent
     notify/mailer: 100% learned, 315 records, 13 errors, last 2026-10-16T13:59:00.183000+00:00, silent

6. Next

   Turn the flows discover found into rules with Suggested rules: they start in shadow mode and page nobody until you promote them. When you need a check no suggestion covers, see Write your own rules.

Log tiers

Kinro works with any logs. discover gives every service a tier from what its lines carry. A higher tier makes the results more exact, but no tier is required: discover, timeline and the generic monitors work at L0.

A service is L2 when at least half of its records are contract lines, L1 when at least half are structured and it has id fields, and L0 otherwise. The report says what the next tier would add for each service, for example:

next tier for 1 L0 services: Structured lines (JSON or logfmt) with ID fields: fields are read as they are instead of being guessed from the text, so entity keys and cross-service links no longer depend on message wording.
next tier for 2 L1 services: The Kinro contract (ContractFormatter, one line in the logging setup): explicit event names and keys instead of mined templates, so there are no template or join mistakes.

Headers Kinro reads by itself: ISO timestamps, [ts +zone], access-log [16/Oct/2026:…], celery [ts,ms: LEVEL/…], syslog, and a timestamp later in the line; the level word; JSON lines, GELF, logfmt, Docker json-file and text. Tracebacks and indented dumps join the line they belong to.

Discover

kinro discover reads a folder of log files, or samples a live Loki or Graylog, and reports what it finds without rules, without the SDK and without any required log format. It only reads the logs. It writes three files to --out and prints a short summary.

$ kinro discover --logs ./logs --out ./discovery
$ kinro discover --loki https://loki.internal:3100 --since 24h --out ./discovery
$ kinro discover --graylog https://graylog.internal:9000 --query 'environment:prod' --since 24h --out ./discovery

The commands on this page show the kinro command of the image. With Docker, run them as docker run --rm --user "$(id -u):$(id -g)" -v ... kybalda/utyg:0.1.3 discover ..., as in the Quickstart.

What it finds

1. Records. Files are read recursively (hidden and binary files skipped, .gz read), with terminal colours removed. A line without a timestamp takes the previous one.
2. Templates. Variables are masked (uuids, hex, numbers, quoted strings, URL queries, IPs, timestamps), then a Drain tree groups lines into templates with stable ids. Access lines are grouped per route.
3. Fields and ids. Name and value pairs from JSON keys, k=v, k: v, URL path segments and context tags. Memory stays fixed on any volume: each field keeps a bounded sample of its values.
4. Links and keys. Fields of different services that share values. A link is confirmed with at least --min-shared shared values, containment of at least --min-containment on one side, id-like values on both sides and more overlap than chance. Weaker candidates are listed as unconfirmed with the reason. Fields joined by confirmed links form keys (k1, k2, ...).
5. Flows. For each key, the events of every entity in time order across services: transitions with counts, P(next | prev), p50/p95/p99 of the time between them, and the most common paths.
6. Findings. Top error templates; error bursts (a minute at or above both 10 errors and 10 times the service's usual error rate, with a peak at least 3 times its usual error minute; hot minutes within 30 minutes merge); repeated failures (one entity hitting one failure at least --retry-min times); silent gaps (longer than both 30 minutes and 10 times the service's median pause); secrets and personal data.

Options for files

From Loki or Graylog

No files and no project file are needed: discover can sample the log store you already have, with read-only access. The token is always read from an environment variable (or .env); a flag only names that variable, so the token never appears in shell history or the process list.

$ export LOKI_URL=https://loki.internal:3100 LOKI_TOKEN=...     # LOKI_TENANT for X-Scope-OrgID, if any
$ kinro discover --loki --selector '{namespace="prod"}' --service-label app \
    --since 2026-10-15T00:00Z --until 2026-10-16T00:00Z --sample 20000 --out ./discovery
$ export GRAYLOG_TOKEN=...                                      # an access token of a Reader user
$ kinro discover --graylog https://graylog.internal:9000 --service-field source --fields request_id --out ./discovery

• Sampling. Each service reads every slice from its start until the slice ends or its share of --sample and --max-mb is used; a quiet slice passes its unused share on. A small service is read whole, a busy one in evenly spread stretches. The report's Source section shows what share of the window each service covers; a sampled service gets no silent-gap findings.
• Hints. When Loki has them, index/volume, detected_fields and patterns add sizes, field cardinalities and pattern counts. A missing endpoint is marked unavailable and not asked again.
• Read-only by construction. The Loki client sends only GET to query_range, labels, label/<name>/values, index/volume, detected_fields and patterns; the Graylog client only POST /api/search/messages (Graylog 5.1+). Any other request raises before it is sent.

Exit codes: 0 done, 1 the store cannot be read, 2 a usage error.

Timeline

Type an id, such as a job id, a request id or an entity id, and see its events across all services in time order, with no rules. timeline reads the index.sqlite that discover wrote, read-only.

$ kinro timeline cf9df4e64fe6 --index ./discovery
kinro timeline cf9df4e64fe6: 2 events in 2 services (edge/api, notify/mailer), 2026-10-16T08:59:16.502Z .. 2026-10-16T08:59:17.988Z
key k1 job (confirmed link across services)
2026-10-16T08:59:16.502Z  edge/api       info     job submitted  [job_ref]
2026-10-16T08:59:17.988Z  notify/mailer  info     job cf9df4e64fe6 delivered  [job]
related ids in the same records (keys followed one hop)
  95513  edge/api:latency_us  1 record
  cb8389fbea81ad63  edge/api:trace  1 record

The first event comes from edge/api, whose line names the job in a job_ref field; discover confirmed that job_ref and the mailer's job are one key, so both show up. Each event line shows the time, service, level, a cleaned excerpt and the field that matched.

Exit codes: 0 found; 1 not found, with the closest indexed values and a note when a cap of the index was hit; 2 a usage error or a missing index.

$ kinro timeline 9d37ff01fe8 --index ./discovery
kinro timeline 9d37ff01fe8: not found in the index
  closest indexed values: 9d37ff01fe80

Watching without rules

kinro run without --project is zero-config mode. It reads every service of a Loki, a Graylog and/or a log file glob named in KINRO_* variables, mines a template for every line, and runs four generic monitors on event time. What they find is an insight: kept in the state, listed in a daily digest, and delivered at once only for the monitors you opt into. Custom rules (--project) work as before; the two modes do not mix in one process.

$ kinro check                  # validates the KINRO_* variables offline
$ kinro run                    # learn, then report
$ kinro status                 # monitors, learning progress, insight counts, next digest ($KINRO_STATE_PATH)
$ docker run -d --init --env-file .env -v kinro-state:/state -e KINRO_STATE_PATH=/state/kinro.db kybalda/utyg:0.1.3 run

The image's default command needs a project file, so name run explicitly for zero-config mode.

Monitors

Learning, digest and paging

• Learning. For KINRO_LEARN_HOURS (24) from its first record, a service's monitors only learn; what they would report is counted as learning shadow insights in kinro status and the digest.
• Digest. One message a day at KINRO_DIGEST_AT (09:00 UTC): new insights grouped by monitor, ongoing ones (silent services, running spikes and loops), learning shadow counts and rate-limited ones. Its id derives from the day, so a restart never sends it twice.
• Paging. Only the monitors listed in KINRO_PAGE (for example error_spike,silent_service, or all) are delivered at once. The default is none: everything waits for the digest. Source-health meta-alerts (kinro.source_stalled, kinro.source_gap, kinro.parse_errors) are always delivered at once, also while learning.
• Channels. The same as with rules: Telegram and Slack when their variables are set, the file at KINRO_ALERTS_PATH, and the log.

[KINRO INSIGHT] repeated_failure: notify/mailer
notify/mailer: job=****fe80 hit "job <hex> failed: upstream unavailable" 5 times in 4m (a retry loop?)
at: 2026-10-16T12:52:36.488000+00:00, detected: 2026-10-16T12:56:36.488000+00:00
template: {"id": "t:a34f17d99e", "template": "job <hex> failed: upstream unavailable", "example": "job ****fe80 failed: upstream unavailable", "total": 5, "errors": 5}
field: "job"
value: "****fe80"
hits: 5
window: ["2026-10-16T12:52:36.488000+00:00", "2026-10-16T12:56:36.488000+00:00"]
window_limit_minutes: 60.0
retry_min: 5
alert_id: 062fc30ce6e15caa

Environment

Sources; several can be combined, each one reading many services:

Monitors and delivery:

How lines are read

Contract lines keep their event names, keys, inst and seq. Any other line goes through the discover header parser, the template miner (the event becomes t:<template id>) and the field extractor (ids: uuids, numbers, hex, tokens; never emails or fields named like personal data). A line's severity is its level, raised to error by a traceback, a 5xx status or a failure word when the line has no level. Every evidence string is redacted (<secret>, <email>, <phone>) and ids are masked (****1234) before they reach the state, the outbox or a notifier.

• Restarts. Templates, counts, baselines, pauses and retry counters commit with the source cursors in one SQLite transaction, so after a restart the same line gets the same template id. Each insight has a deterministic alert_id, so a restart never duplicates one.
• Bounded memory. 300 services (the rest count as <other services>), 1000 templates per service, 20,000 tracked ids, 24 h of per-minute error counts per service.
• Measured noise. On synthetic healthy services with every monitor paging, 3 days and 48k lines gave 0 insights, and 14 days and 223k lines gave 1: a planted rare error that first appears on day 3.
• Known limits. A service learned on weekdays may be reported silent over a weekend; file lines are records one by one; stores are read back only within KINRO_LOOKBACK.

Suggested rules

kinro suggest turns a discover --out folder into a project file you can review, check and run. Its rules are the built-ins chain, no_duplicates and ordering, with thresholds learned from your data, and every rule carries its evidence as a comment. Nothing is written unless the file passes validation.

$ kinro suggest --from ./discovery --out project.py --logs-root /logs
suggest: 2 services (1 left out), 1 keys (job), 4 rules (1 chain, 2 no_duplicates, 1 ordering), 3 candidates skipped (support: 3); 300 entities, 0 ambiguous
wrote project.py: 2 services, 4 shadow rules; validated. Next: kinro check --project project.py, then kinro replay --project project.py
$ kinro check --project project.py
suggested: 4 rules, 2 services, 0 dbs; 0 errors, 0 warnings
$ kinro replay --project project.py           # shadow violations print as SHADOW <rule> ...
SHADOW chain_job_submitted_2c772f job=9d37ff01fe80 008fe6efee80e596: missing step notify/mailer/job_delivered within 0:00:11
replay: 1 violations (1 shadow), 0 meta-alerts, 0 late follow-ups; 600 observations, 0 late, 0 rule errors

The one shadow violation is the job that kept failing: it was delivered only after its retry loop, long after the learned 11 seconds.

A part of the file it wrote:

project.service(
    'notify/mailer',
    LogSource('/logs/notify/mailer-*.log', parser=TemplateParser(
        {
            # t733289dd8b, 300 records, rule step; parser check 100.0% of 300 lines
            'job_delivered': 'job <*> delivered',
        },
        fields=['job'],
    )),
    keys={
        'job': Field('job', normalize=str.lower),
    },
    # Discovery found secrets or personal data under these names.
    redact=['email'],
)

project.add(
    # 'job_submitted' (edge/api) is followed by 'job_delivered' (notify/mailer) within 11s: 298 of 299 flows by job
    # (cross-service key, confidence 0.997); latency to the slowest step p50 1.5s, p95 1.65s, p99 1.75s; within =
    # p99 x 3 + 5s, rounded up. Examples ****b10e, ****960e, ****01a2; source 3 files under /logs.
    chain(
        'chain_job_submitted_2c772f',
        key='job',
        steps=[API_JOB_SUBMITTED, MAILER_JOB_DELIVERED],
        within=timedelta(seconds=11),
        shadow=True,
    ),
)

What goes into the file

• Services. One project.service(...) per discovered service that a rule needs: LogSource(<glob>) for files (a pod hash becomes *; --logs-root says where Kinro finds the files), LokiLogs(LokiConfig(selector=...), url_env='LOKI_URL') or GraylogLogs(GraylogConfig(query=...), url_env='GRAYLOG_URL', token_env='GRAYLOG_TOKEN') when discover read a store. Contract services keep StandardParser; every other one gets a TemplateParser with the discovered templates its rules use. Services no rule needs are left out, as a comment.
• Keys. One key per confirmed link group, named after its fields. Text and hex ids are read with Field(name, normalize=str.lower), since discover matched them case-insensitively. Fields where discover found secrets or personal data go to redact=[...].
• chain when every step follows the first in at least 98% of at least 50 flows; within = p99 latency to the slowest step × 3 + 5 s, rounded up to whole seconds. Chains slower than 1 h stay candidates.
• no_duplicates when fewer than 0.1% of entities repeat an event within 1 h (else the longest of 30, 10, 5 or 1 min that passes).
• ordering for consecutive chain steps in different services when at most 0.1% of flows invert them beyond 1 s; only for services whose lines carry their own time.
• Parser check. Each template is run against the indexed lines of its service; a template that does not map back to itself in 99% of them drops its rules to candidates.
• Candidates. Rules below a threshold are listed commented out, each with its reason. Repeated failures come as a commented custom RepeatedFailure rule, database checks as commented db_invariant/status_desync examples.

Rule ids are a slug plus a short hash of the rule's kind, key and events, so a second run on similar data gives the same ids and two runs can be compared.

Exit codes: 0 written, 1 the suggested project does not validate (nothing is written), 2 a usage error.

Shadow rules and the shadow channel

Every suggested rule has shadow=True. It runs as usual, but its violations go only to the shadow channel, which writes the alerts file (alerts_path) and the log (SHADOW ALERT, titled [KINRO SHADOW]) and never Telegram or Slack. They keep their alert_id and are counted in kinro status:

shadow violations (shadow channel only): chain_job_submitted_2c772f=1

Any rule can be a shadow rule: chain(..., shadow=True), MyRule('id', shadow=True), or shadow = True on the class.

Promote a rule

1. Run it in shadow

   Mount the folder with project.py at /app and the logs at --logs-root, as in Write your own rules. If a zero-config Kinro keeps running, give the project its own state file with --state-path: two processes must never share one.

2. Review

   Read the shadow alerts in the alerts file for a few days, as in the Shadow week. Check a changed threshold with kinro replay.

3. Promote

   Delete the rule's shadow=True, run kinro check, and restart. From then on its alerts go to every channel. Uncomment a candidate the same way to try it in shadow.

Write your own rules

Rules are optional. Write them when a flow needs a check that no suggestion covers: a database comparison, a payload field, a promise only you know. Start from the file kinro suggest wrote, or from the template kinro init writes. This walks through a two-service flow: orders logs to Loki, billing writes a legacy text file, and Kinro alerts when a created order is not paid within 15 minutes.

1. Install the SDK

   Rules and project files need only the kinro-sdk package: no engine and no dependencies, with types for the IDE, under the Apache License 2.0. The engine runs as the Kinro Docker image.

   It is on PyPI, so one command is enough:

   uv pip

   $ uv add kinro-sdk

   Pin the version your Kinro image runs, for example kinro-sdk==0.1.3. Without internet access, install the wheel from your Kinro release or your private index instead:

   $ pip install kinro_sdk-0.1.3-py3-none-any.whl
   $ pip install kinro-sdk --index-url https://<your-private-index>/simple

   To run rule tests with RuleHarness locally, also install the kinro engine package; it pulls in the matching SDK.

2. Describe the project

   A project file is a Python module with a module-level project.

   from datetime import timedelta
   
   from kinro.sdk import Field, LogSource, LokiConfig, LokiLogs, Project, RegexParser, chain, on
   
   project = Project('shop', state_path='/state/kinro.db', alerts_path='/state/alerts.jsonl')
   
   project.service(
       'orders',
       LokiLogs(LokiConfig(selector='{service="orders"}')),
       keys={'order': Field('uid', normalize=str.lower)},
   )
   project.service(
       'billing',
       LogSource('/logs/billing/billing.log', parser=RegexParser(
           r'^(?P<ts>\S+ \S+) (?P<level>\w+) (?P<event>payment captured) ext=(?P<external_id>\w+)$',
           events={'payment captured': 'payment.captured'},
           ts_format='%Y-%m-%d %H:%M:%S,%f',
       )),
       keys={'order': Field('external_id', normalize=str.lower)},
   )
   
   project.add(
       chain('order_paid', key='order',
             steps=[on('orders', 'order.created'), on('billing', 'payment.captured')],
             within=timedelta(minutes=15)),
   )

3. Create a rules folder

   With the image from the Quickstart, create the rules folder from the template built into it. It holds project.py with working examples, rules/, parsers/, tests/, compose.yaml, .env.example and a Dockerfile. Create the folder yourself first, so it belongs to your user.

   $ mkdir shop-rules
   $ docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/shop-rules:/out" kybalda/utyg:0.1.3 init /out
   $ cd shop-rules

4. Add compose.yaml and .env

   Keep everything in one rules folder and mount the whole folder read-only at /app. This is the recommended setup: Kinro puts /app on the import path, so rules/ and parsers/ import, and a new rule file needs no change to compose.yaml. The log directory is mounted read-only too; the state lives in a volume.

   shop-rules/
     project.py        # services, databases and the rules to run
     rules/            # custom rules, one module per business flow
     parsers/          # custom parsers for legacy log formats
     compose.yaml
     .env              # secrets; keep it out of git

   services:
     kinro:
       image: kybalda/utyg:0.1.3
       env_file: .env
       init: true                  # SIGTERM reaches Kinro, which commits before exit
       restart: unless-stopped
       volumes:
         - kinro-state:/state
         - ./:/app:ro                # the rules folder: project.py, rules/, parsers/
         - /var/log/billing:/logs/billing:ro
   volumes:
     kinro-state:

   LOKI_URL=https://loki.internal
   # TELEGRAM_BOT_TOKEN=...
   # TELEGRAM_CHAT_ID=...

   Updating rules. Edit or add files in the folder, check, then restart. Kinro loads rules at start, so the restart applies them; state and pending deadlines survive it. Bump a rule's version when its logic changes, which resets that rule's stored state. Keep the folder in git, so every rule change is reviewed and can be rolled back.

   $ docker compose run --rm kinro check --project /app/project.py
   $ docker compose restart kinro

5. Validate it

   check validates rules, sources and environment without connecting anywhere. The image's entrypoint is kinro, so the subcommand follows the service name.

   $ docker compose run --rm kinro check --project /app/project.py
   shop: 1 rules, 2 services, 0 dbs; 0 errors, 0 warnings

6. Start it

   By default the container runs kinro run --project /app/project.py. Without Telegram or Slack settings, alerts go to the log and to alerts_path. That is the recommended first week; see Shadow week. Rules with shadow=True stay on the shadow channel (alerts file and log) even when the chats are set.

   $ docker compose up -d
   $ docker compose logs -f kinro

Log contract

LogContract v1 is the line format Kinro reads natively: one JSON object per line. It is recommended, not required: it is tier L2, the top tier. Services emit it with ContractFormatter. Discover, timeline and the generic monitors read any other format as it is; for rules on such a service, use a parser, such as the TemplateParser that kinro suggest writes.

{"kinro_v": 1, "ts": "2026-10-15T12:00:00.123456+00:00", "svc": "orders", "event": "order.created",
 "level": "INFO", "cid": "0192f0c4-7c1e-7d3a-9b1e-5f2a1c3d4e5f", "keys": {"uid": "A1B2"},
 "data": {"amount": "10.50"}, "msg": "order A1B2 created", "inst": "orders-7f9c:12", "seq": 41}

Correlation keys

Every rule has a key: the business identity it watches, such as order. Each service says which of its fields carries that key. Kinro joins observations in two ways.

• Shared id. Services that log the same cid are joined directly. A service that logs only the cid (shipping below) still lands in the order timeline through another service that logs both.
• Key mapping. When field names differ, map them to one logical key: uid in orders and external_id in billing both mean order. Field(name, normalize=...) fixes case or formatting; normalise every side the same way, or A1B2 and a1b2 stay two keys.

project.service('orders', orders_logs, keys={'order': Field('uid', normalize=str.lower)})
project.service('billing', billing_logs, keys={'order': Field('external_id', normalize=str.lower)})
project.service('shipping', shipping_logs)          # only cid; joined through orders

# A rule keyed by cid sees shipments grouped by correlation id instead of order.
no_duplicates('single_shipment', key='cid', trigger=on('shipping', 'shipment.created'),
              within=timedelta(hours=1))

Field lookup checks cid, then keys, then the payload, so a regex group that is not listed in keys can still be mapped.

Event time and deadlines

Rules run on when things happened, not when log lines arrived. Each service has a watermark: the time up to which Kinro has seen everything. It trails the newest event time by max_lateness.

• A deadline fires only after the watermark of every service that could still change the outcome passes it. A slow log pipeline delays alerts but never causes a false one.
• For Loki services, heartbeats from start_heartbeat move the watermark of a quiet producer forward.
• A step that arrives after its deadline already fired produces a follow-up alert <alert_id>#late1.
• When Kinro knows logs were lost (a gap in seq, Loki retention), deadlines that overlap the gap become unverifiable: one kinro.source_gap alert instead of many false ones.

LATENESS = timedelta(seconds=10)    # about 2x the p99 push delay you see in Grafana

# loki() is the helper from the Grafana Loki section below.
project.service('orders', loki('orders'), keys={'order': 'uid'}, max_lateness=LATENESS)

Defaults: max_lateness=5s, poll_interval=0.5s. Every deadline alert waits at least max_lateness.

Built-in rules

Five rules cover the common failures. They are written with the same public SDK as your own rules. kinro suggest writes the first three for you, with learned thresholds.

CREATED = on('orders', 'order.created')
PAID = on('billing', 'payment.captured')
SHIPPED = on('shipping', 'shipment.created')

project.add(
    # Every step arrives within 30 minutes of the first one.
    chain('order_flow', key='order', steps=[CREATED, PAID, SHIPPED], within=timedelta(minutes=30)),

    # At most one payment per order per hour.
    no_duplicates('single_payment', key='order', trigger=PAID, within=timedelta(hours=1)),

    # Payment comes before shipment; 2 s of clock skew is tolerated.
    ordering('paid_before_shipped', key='order', first=PAID, then=SHIPPED, tolerance=timedelta(seconds=2)),

    # The status a service logged is in the database 60 s later.
    status_desync(
        'order_status_in_db', key='order', trigger=on('orders', 'order.status_changed'), field='status',
        db='orders_db', settle=timedelta(seconds=60),
        lookup='SELECT uid AS key, status, updated_at FROM orders WHERE uid = ANY(:keys)',
    ),

    # Rows that must never exist.
    db_invariant(
        'paid_orders_have_amount', key='order', db='orders_db', every=timedelta(minutes=1),
        sql="SELECT uid AS key, amount FROM orders WHERE status = 'paid' AND amount <= 0",
    ),
)

A chain alert reads like missing step shipping/shipment.created within 0:30:00 and names the last step that did arrive as the broken step. Every built-in also takes shadow=True: its violations then go only to the shadow channel.

Rule classes

A rule is a class. It declares its key and triggers, reacts to observations and its own deadlines, keeps state per key, and calls ctx.violate when something is wrong. The engine isolates rules: an exception in one raises kinro.rule_error and never stops the others.

from datetime import timedelta

from kinro.sdk import Observation, Rule, RuleContext, on

REFUNDED = on('payments', 'refund.issued')
TICKET = on('support', 'ticket.created')


class RefundHasTicket(Rule):
    key = 'order'
    version = 1
    triggers = (REFUNDED, TICKET)
    window = timedelta(hours=2)

    def on_observation(self, ctx: RuleContext, obs: Observation) -> None:
        if obs.event == REFUNDED.event:
            ctx.set_deadline('ticket', obs.ts + timedelta(minutes=10), wait_for=['support'])
        elif obs.event == TICKET.event:
            ctx.cancel_deadline('ticket')
            ctx.close()

    def on_deadline(self, ctx: RuleContext, name: str) -> None:
        ctx.violate('refund issued without a support ticket within 10 minutes',
                    expected='support/ticket.created')
        ctx.close()


project.add(RefundHasTicket('refund_has_ticket'))

class StuckInCreated(Rule):
    key = 'order'
    version = 1
    schedules = (Query(
        'orders_db', name='stuck_created', every=timedelta(minutes=5),
        sql="SELECT uid AS key, created_at FROM orders WHERE status = 'created' "
            "AND created_at < now() - interval '30 minutes' LIMIT 100",
    ),)

    def on_observation(self, ctx: RuleContext, obs: Observation) -> None:
        if obs.event == self.schedules[0].row_event:
            ctx.violate('order stuck in created for more than 30 minutes', broken_step=obs)
            ctx.close()

Rule attributes

RuleContext

Hooks: on_observation(ctx, obs), on_deadline(ctx, name) and on_unverifiable(ctx, name), which runs instead of on_deadline when lost logs overlap the deadline window.

Testing rules

RuleHarness drives one rule through the real engine without Loki, files, a database or notifiers. Feed observations, move the watermark, read the violations. It ships with the kinro engine package, not with kinro-sdk.

from datetime import UTC, datetime, timedelta

from kinro.contract.model import FileObsId, Observation
from kinro.testing import RuleHarness

from rules.refunds import RefundHasTicket

T0 = datetime(2026, 10, 16, 12, 0, tzinfo=UTC)


def obs(service: str, event: str, seconds: int, order: str, offset: int) -> Observation:
    return Observation.create(
        obs_id=FileObsId(service, 1, 0, offset), service=service, event=event,
        ts=T0 + timedelta(seconds=seconds), payload={'order': order},
    )


def test_refund_without_ticket_alerts() -> None:
    harness = RuleHarness(RefundHasTicket('refund_has_ticket'))
    harness.feed(obs('payments', 'refund.issued', 0, 'A1B2', 1))
    harness.advance(T0 + timedelta(minutes=11))

    [violation] = harness.violations
    assert violation.message.startswith('refund issued without a support ticket')


def test_ticket_in_time_is_quiet() -> None:
    harness = RuleHarness(RefundHasTicket('refund_has_ticket'))
    harness.feed(obs('payments', 'refund.issued', 0, 'A1B2', 1), obs('support', 'ticket.created', 120, 'A1B2', 2))
    harness.advance(T0 + timedelta(minutes=11))

    assert harness.violations == []

Without a project, the harness maps rule.key from a payload field of the same name in every trigger service. Pass your real project to test with its key mappings. harness.poll(db, name, rows, at) and harness.lookup(...) feed database results.

Emitting from a service

A Python service needs one formatter on its Loki handler and one heartbeat. kinro.contract uses only the standard library.

import logging

import logging_loki
from kinro.contract import ContractFormatter, start_heartbeat

loki = logging_loki.LokiHandler(url=LOKI_PUSH_URL, tags={'service': 'orders'}, version='1')
loki.setFormatter(ContractFormatter('orders'))      # one formatter instance per handler
logging.getLogger().addHandler(loki)
start_heartbeat(logging.getLogger())                # every 2 s, ignores LOG_LEVEL

Then name the business event in the log calls the service already makes:

logger.info('order %s created', uid, extra={
    'event': 'order.created',
    'cid': cid,
    'keys': {'uid': uid},
    'data': {'amount': str(amount)},
})

Records without event keep their usual JSON shape and are ignored by Kinro. Services in other languages write the same JSON line; the contract is the whole interface.

Grafana Loki

LokiLogs reads a stream selector through /loki/api/v1/query_range only, with a read-only token. One client per Loki URL and credentials enforces max_qps.

def loki(service: str) -> LokiLogs:
    return LokiLogs(
        LokiConfig(
            selector=f'{{service="{service}", environment="production"}}',
            ingest_lateness=timedelta(seconds=10),
            retention=timedelta(days=7),
            start_lookback=timedelta(minutes=15),
        ),
        url_env='LOKI_URL',
        token_env='LOKI_TOKEN',
        tenant_env='LOKI_TENANT',
        max_qps=2.0,
    )

Graylog

Services that log to Graylog over GELF (UDP, TCP or HTTP) are read with GraylogLogs. Kinro calls only the search API POST /api/search/messages with a read-only access token. It never writes to Graylog and never receives GELF itself. Requires Graylog 5.1 or newer.

In the service

graypy already sends every extra={...} item as a GELF field, so the service keeps its handler. Add one filter that stamps inst and seq, and the heartbeat.

import logging

import graypy
from kinro.contract import GelfContractFilter, start_heartbeat

gelf = graypy.GELFHTTPHandler('graylog.internal', 12201, timeout=5)   # or GELFUDPHandler / GELFTCPHandler
gelf.addFilter(GelfContractFilter())                                  # one instance per handler
logging.getLogger().addHandler(gelf)
start_heartbeat(logging.getLogger())

logger.info('order %s created', uid, extra={
    'event': 'order.created', 'cid': cid, 'key_uid': uid, 'amount': str(amount),
})

In the project

orders = GraylogLogs(
    GraylogConfig(
        query='facility:orders',               # Lucene query that selects the service
        streams=('5f1e2d3c4b5a69788796a5b4',),  # stream ids; empty means every readable stream
        fields=('amount', 'status'),           # payload fields any rule reads
        ingest_lateness=timedelta(seconds=10),
    ),
    url_env='GRAYLOG_URL', token_env='GRAYLOG_TOKEN', max_qps=2,
)
project.service('orders', orders, keys={'order': 'uid'}, max_lateness=timedelta(seconds=5))

• Lost messages. GELF over UDP drops datagrams silently. A missing seq raises kinro.source_gap, and deadlines over the gap are not alerted. Use the HTTP or TCP handler where loss matters.
• Equal timestamps. Graylog stores milliseconds, so many messages share one. Kinro pages by time and tells messages apart by gl2_message_id, which survives index rotation and restarts. More than 10 000 messages in one millisecond raise kinro.graylog_page_overflow.
• Indexing delay. ingest_lateness must cover the time until a message becomes searchable: transport, output batching and the index refresh, about 2 s on a default setup.
• Retention. Set retention to the index set retention, so a range Graylog deleted before Kinro read it raises kinro.source_gap.
• The access token belongs to a user with the Reader role and read access to the searched streams only.

Log files

LogSource tails files matched by a glob. It never re-reads a file in full, keeps its own cursor, and survives rotation by rename or copytruncate without losing lines.

LogSource(
    '/logs/billing/*.log',
    parser=RegexParser(...),
    rotated='{path}.[0-9]*',        # where rotated copies appear; *.gz never matches
    start_at='end',                 # files that exist on first deploy: 'end' or 'beginning'
)

Mount the directory read-only. A service that only writes files must run on the Kinro host.

volumes:
  - /var/log/billing:/logs/billing:ro

Custom parsers

Services you cannot change still count. Pick the parser that fits their format.

RegexParser(
    r'^(?P<ts>\S+ \S+) (?P<level>\w+) \[billing\] (?P<event>payment captured|payment failed)'
    r' ext=(?P<external_id>\w+) amount=(?P<amount>[\d.]+) card=(?P<card>\S+)$',
    events={'payment captured': 'payment.captured', 'payment failed': 'payment.failed'},
    ts_format='%Y-%m-%d %H:%M:%S,%f',
    tz='Europe/Kyiv',               # naive timestamps are read in this zone
)

2026-10-16 15:00:05,532 INFO [billing] payment captured ext=A1B2 amount=10.50 card=4111111111111111

{"service": "billing", "event": "payment.captured", "ts": "2026-10-16T12:00:05.532+00:00",
 "payload": {"external_id": "A1B2", "amount": "10.50", "card": "[redacted]"}}

TemplateJsonParser([
    MessageRule(r'order (?P<uid>\w+) created', event='order.created', keys=('uid',)),
    MessageRule(r'order (?P<uid>\w+) cancelled: (?P<reason>.+)', event='order.cancelled', keys=('uid',)),
])

TemplateParser(
    {
        'job_submitted': 'job submitted',
        'job_failed': 'job <hex> failed: upstream unavailable',
    },
    fields=['job_ref', 'job'],
)

import csv
from datetime import datetime

from kinro.contract.model import Observation
from kinro.parsers.base import LineMeta, ParseError, Parser


class WarehouseCsvParser(Parser):
    """Lines like `2026-10-16T12:00:09Z;shipped;A1B2;dhl`."""

    def _parse(self, line: str, meta: LineMeta) -> Observation | None:
        row = next(csv.reader([line], delimiter=';'))
        if len(row) != 4:
            raise ParseError(f'expected 4 columns, got {len(row)}')
        ts, status, order, carrier = row
        if status != 'shipped':
            return None
        return Observation.create(
            obs_id=meta.obs_id, service=meta.service, event='shipment.created',
            ts=datetime.fromisoformat(ts), keys={'order': order}, payload={'carrier': carrier},
        )

PostgreSQL

Rules can query a database on a schedule (Query) or look up specific keys on demand (Lookup). Kinro connects with a role that can only read.

CREATE ROLE kinro_ro LOGIN PASSWORD '...';
GRANT CONNECT ON DATABASE orders TO kinro_ro;
GRANT USAGE ON SCHEMA public TO kinro_ro;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO kinro_ro;
ALTER ROLE kinro_ro SET default_transaction_read_only = on;
ALTER ROLE kinro_ro SET statement_timeout = '5s';
ALTER ROLE kinro_ro SET lock_timeout = '1s';
ALTER ROLE kinro_ro SET idle_in_transaction_session_timeout = '10s';

project.db('orders_db', url_env='ORDERS_DB_URL', min_interval=timedelta(seconds=30), pgbouncer=True)

• Every statement must be a single SELECT or WITH; validate rejects anything else.
• Each query returns a key column. Rows arrive to rules as observations with event db.query:<name>; for lookups, db.lookup:<name> and db.lookup_missing:<name>.
• A failed poll changes nothing: rules never see an empty result that was really an error.
• Point Kinro at a replica when you have one.

MySQL

MySQL 8.x works through the same project.db(...) call. Kinro picks the dialect from the URL, so rules do not change except for lookups.

CREATE USER 'kinro_ro'@'%' IDENTIFIED BY '...';
GRANT SELECT ON shop.* TO 'kinro_ro'@'%';
ALTER USER 'kinro_ro'@'%' WITH MAX_USER_CONNECTIONS 2;

SHOP_DB_URL=mysql://kinro_ro:password@db.internal:3306/shop

project.db('shop_db', url_env='SHOP_DB_URL')

status_desync(
    'order_status', key='order', trigger=on('orders', 'order.status_changed'), field='status',
    db='shop_db', settle=timedelta(seconds=30),
    lookup='SELECT uid AS `key`, status, updated_at FROM orders WHERE uid IN :keys',
)

• Lookups use IN :keys. MySQL has no arrays, so = ANY(:keys) works only on Postgres. IN :keys works on both.
• key is reserved. Write AS `key` with backticks. kinro check reports both mistakes.
• Read-only session. On connect Kinro sets the session read-only, max_execution_time to 5 s, lock waits to 1 s and the time zone to UTC. Only the SELECT grant truly stops writes, so grant nothing else.
• Times. DATETIME values have no zone and are read as UTC. If the application stores local time, convert it in the query with CONVERT_TZ.
• Values. BOOLEAN comes back as 0/1 and JSON columns as strings; cast in the query when a rule compares them.
• MariaDB is not supported, and pgbouncer=True applies to Postgres only.

Alert format and channels

Every alert carries the rule, the key, the cross-service timeline and the broken step, marked with >>.

[KINRO] order_flow v1
alert_id: 7f3c9e21a4d0b8e6
key: order=A1B2
missing step shipping/shipment.created within 0:30:00
broken step: 2026-10-16T12:00:05Z billing payment.captured external_id=a1b2 {"amount": "10.50"}
expected: "shipment.created"
deadline: complete @ 2026-10-16T12:30:02Z
detected: 2026-10-16T12:30:09Z
timeline (2):
   2026-10-16T12:00:02Z orders order.created cid=0192f0c4 uid=A1B2 {"amount": "10.50"}
>> 2026-10-16T12:00:05Z billing payment.captured external_id=a1b2 {"amount": "10.50"}

Alerts are written to an outbox in the same transaction as the state that produced them, then delivered. A restart never loses an alert. Delivery is at-least-once: a crash between sending and recording can resend an alert, with the same alert_id. The file channel skips ids it already wrote. A channel that answers with a permanent error marks the alert failed instead of retrying forever.

Meta-alerts

Alerts about Kinro itself start with kinro. and show up as [KINRO META].

Redaction

List payload fields that must never leave Kinro. Their values become [redacted] as soon as a line is read, before storage, rules and alerts.

# loki() is the helper from the Grafana Loki section.
project.service('orders', loki('orders'), keys={'order': 'uid'}, redact=['email', 'phone', 'address'])
project.service('billing', billing_file, keys={'order': Field('external_id')}, redact=['card'])

• Names match top-level payload fields exactly. A dict under a redacted name is replaced as a whole.
• cid and keys are identity and are never redacted, so keep personal data out of them. validate rejects a redacted key field or a redacted status_desync field.
• Database rows are not redacted. Select only the columns a rule needs.
• Discover, timeline, suggest and the zero-config monitors mask secrets and personal data on their own, with no redact list; kinro suggest adds the fields where it found them to redact=[...].

CLI reference

kinro always needs a subcommand; without one it prints usage and exits 2. In the image the entrypoint is kinro, so the subcommand follows the image name (docker run ... kybalda/utyg:0.1.3 discover ...) or the compose service name (docker compose run --rm kinro check ...).

$ docker run --rm --user "$(id -u):$(id -g)" -v "$PWD/logs:/logs:ro" -v "$PWD/discovery:/out" kybalda/utyg:0.1.3 discover --logs /logs --out /out
$ docker run --rm -v "$PWD/discovery:/out:ro" kybalda/utyg:0.1.3 timeline 9d37ff01fe80 --index /out
$ docker run --rm --env-file .env kybalda/utyg:0.1.3 check                # zero-config env
$ docker exec kinro kinro status                                          # zero-config container
$ docker compose run --rm kinro check --project /app/project.py          # validate; connects nowhere
$ docker compose up -d                                                    # run: watch and alert
$ docker compose run --rm kinro replay --project /app/project.py --since 3600 --json   # past logs, delivers nothing
$ docker compose run --rm kinro replay --project /app/project.py --since 3600 --db     # also run DB checks live
$ docker compose run --rm kinro status --project /app/project.py --json   # lag, counters, outbox
$ docker compose run --rm kinro status --state /state/kinro.db --alert 7f3c9e21a4d0b8e6
$ docker compose run --rm kinro --help

Set --commit-interval and --stats-interval with command: in compose.yaml. Exit codes: 0 ok, 1 failed (check errors, a crash, a store that cannot be read, an id not found, a suggestion that does not validate), 2 configuration or usage error.

Deployment

Run exactly one Kinro process per project, next to its state volume. Services on any host keep shipping logs to Loki. Zero-config mode needs no project and no compose file: one docker run ... kybalda/utyg:0.1.3 run with the KINRO_* env and a /state volume, as in the Quickstart. With rules, use the compose file below.

host A: orders, shipping ──push──> Grafana Loki <──query_range (read-only)── Kinro
host B: billing ──> /var/log/billing/*.log ──:ro mount──────────────────────┘  │
                         Postgres (kinro_ro, replica or PgBouncer) <──SELECT───┘
                         state: SQLite WAL on a volume (/state/kinro.db)

services:
  kinro:
    image: kybalda/utyg:0.1.3
    env_file: .env
    init: true                  # SIGTERM reaches Kinro, which commits before exit
    restart: unless-stopped
    volumes:
      - kinro-state:/state
      - /var/log/billing:/logs/billing:ro
      - ./:/app:ro                # the rules folder: project.py, rules/, parsers/
volumes:
  kinro-state:

Sizing. About 2 KB of memory per live observation, kept for the rule window. RSS ≈ 2 KB × lines per second × window: 50 lines/s over 30 minutes is about 180 MB.

Monitoring

Every --stats-interval seconds (default 60) Kinro logs one line:

runtime stats: {"lag_seconds": {"orders": 0.8, "billing": 0.4, "orders_db": 12.1},
  "late_observations": 0, "rule_errors": 0, "desync_unverifiable": 0, "chain_unverifiable": 0,
  "link_conflicts": 0, "pending_keys": 1840, "db_queries": 96, "rotation_gaps": 0, "parse_errors": 0}

Healthy steady state: lag stays near max_lateness plus the poll interval; rule_errors, rotation_gaps, link_conflicts and both *_unverifiable counters stay flat; pending_keys stays below max_pending_keys (default 200 000).

kinro status shows lag, the persisted counters, outbox statuses and row counts. Two more parts appear when they apply:

• Shadow rules. shadow violations (shadow channel only): <rule>=<n> per rule. These alerts went only to the shadow channel.
• Zero-config. Services and their learning progress, the paged monitors, next and last digest, insight counts per monitor (and how many were paged), learning shadow and rate-limited counts, and per service its records, errors, last record and whether it is silent or in an error spike.

Shadow week

1. Deploy quietly

   Set alerts_path and leave out the Telegram and Slack variables. Alerts go only to the file and the log. To keep some rules quiet while others page, give them shadow=True: their violations go only to the shadow channel, which writes the same file and the log. Suggested rules start this way. In zero-config mode, leave KINRO_PAGE empty: every insight waits for the digest.

2. Review daily

   Mark each alert as a real bug or noise. Late follow-ups (#late<n>) belong to their alert and are not counted.

   $ jq -r 'select(.alert_id | contains("#late") | not) | [.delivered_at[0:10], .payload.rule_id] | @tsv' \
       /state/alerts.jsonl | sort | uniq -c

3. Tune

   Target: at most 3 false alerts a day over 7 days, no day above 6. Adjust within, settle or max_lateness, and check each change with kinro replay.

4. Go live

   Add the Telegram or Slack variables and restart. Promote a shadow rule by deleting its shadow=True; page a monitor by adding it to KINRO_PAGE. Keep alerts_path as an audit trail.

Recovery

• On start Kinro runs PRAGMA quick_check. A damaged state file stops it with a clear log line; it never runs on broken state.
• Back up the state with the SQLite backup API or after a clean stop; the -wal and -shm files belong to it.
• Deleting the state starts fresh from start_at and start_lookback: pending deadlines are lost and recent alerts may repeat once.
• Disk full or a locked file pauses commits and retries the same snapshot; nothing is applied twice.